nodeos hangs on "terminate called after throwing an instance of 'Serialization::FatalSerializationException'"

[the-smooth-operator]

Since the spring upgrade, we have observed that our nodes suddenly stop responding to RPC calls and stop emitting normal logs to only show terminate called after throwing an instance of 'Serialization::FatalSerializationException' each ~30minutes:

...
debug 2024-10-02T04:20:20.569 nodeos    controller.cpp:4612           clear_expired_input_ ] removed 19 expired transactions of the 3761 input dedup list, pending block time 2024-10-02T01:44:12.500
debug 2024-10-02T04:20:20.585 nodeos    controller.cpp:4612           clear_expired_input_ ] removed 0 expired transactions of the 3750 input dedup list, pending block time 2024-10-02T01:44:13.000
debug 2024-10-02T04:20:20.611 nodeos    controller.cpp:4612           clear_expired_input_ ] removed 30 expired transactions of the 3759 input dedup list, pending block time 2024-10-02T01:44:13.500
debug 2024-10-02T04:20:20.636 net-1     net_plugin.cpp:2244           sync_timeout         ] ["eosn-eos-seed171:9876 - c511972" - 1 209.249.216.152:9876] sync timeout
terminate called after throwing an instance of 'Serialization::FatalSerializationException'
terminate called after throwing an instance of 'Serialization::FatalSerializationException'
terminate called after throwing an instance of 'Serialization::FatalSerializationException'
terminate called after throwing an instance of 'Serialization::FatalSerializationException'

Furthermore, when restarting the process (it restarts gracefully) it always comes up with a corrupted state, so we need to start it from a snapshot, or perform a long replay operation.

error 2024-10-02T07:32:37.800 nodeos    main.cpp:224                  main                 ] 3020000 fork_database_exception: Fork database exception
No existing fork database despite existing chain state. Replay required.
    {}
    nodeos  controller.cpp:1705 replay

    {}
    nodeos  chain_plugin.cpp:1159 plugin_startup

While the process is "stuck" there's network activity in the host and the process is still consuming it's memory and using CPU cycles. However disk operations drop to 0.