failed to open existing databases: dirty state: gossip-294520

[YuXiaoCoder]

** Info
OS：Linux Docker
Sonic Version：1.2.1-e

Describe the bug
Using snapshot (https://download.fantom.network/mainnet-294520-validator.g) loading, the service can be started normally, but after shutting down the service normally and starting it again, it reports that there is dirty data, resulting in the service not being able to be started

To Reproduce
Steps to reproduce the behavior:

1. wget -O /mnt/ftmmain/snapshot/mainnet.g https://download.fantom.network/mainnet-294520-validator.g
2. /opt/ftmmain/core/sonictool --datadir=/mnt/ftmmain/node --cache=16384 genesis /mnt/ftmmain/snapshot/mainnet.g
3. /opt/ftmmain/core/sonicd --datadir=/mnt/ftmmain/node --cache=8192 --mode=rpc --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.corsdomain=* --http.api=admin,eth,web3,net,ftm,txpool,abft,dag --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=admin,eth,web3,net,ftm,txpool,abft,dag --rpc.gascap=0 --rpc.txfeecap=0 --port=30303
4. Normal shutdown of services
5. /opt/ftmmain/core/sonicd --datadir=/mnt/ftmmain/node --cache=8192 --mode=rpc --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.corsdomain=* --http.api=admin,eth,web3,net,ftm,txpool,abft,dag --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=admin,eth,web3,net,ftm,txpool,abft,dag --rpc.gascap=0 --rpc.txfeecap=0 --port=30303

Expected behavior
Services can be started and stopped, and data can be synchronised properly after startup

Screenshots

INFO [07-25|19:30:59.654] New block                                index=86112128 id=294520:232:62af99 gas_used=2,459,933 txs=5/0 age=5d23h5m t=382.612ms
INFO [07-25|19:30:59.661] New block                                index=86112129 id=294520:247:940308 gas_used=984,975   txs=1/0 age=5d23h5m t=6.019ms
INFO [07-25|19:30:59.754] New block                                index=86112130 id=294520:262:ed6ff8 gas_used=737,979   txs=5/0 age=5d23h5m t=78.691ms
INFO [07-25|19:30:59.755] New block                                index=86112131 id=294520:271:2a8f5a gas_used=21000     txs=1/0 age=5d23h5m t="962.896µs"
INFO [07-25|19:30:59.964] New block                                index=86112132 id=294520:311:7bf01d gas_used=1,968,667 txs=7/0 age=5d23h5m t=193.471ms
INFO [07-25|19:31:00.245] New block                                index=86112133 id=294520:340:7a92f5 gas_used=3,408,249 txs=5/0 age=5d23h5m t=271.863ms
INFO [07-25|19:31:00.248] New block                                index=86112134 id=294520:349:0eabc1 gas_used=716,249   txs=1/0 age=5d23h5m t=3.528ms
INFO [07-25|19:31:00.672] Got interrupt, shutting down...
INFO [07-25|19:31:00.673] HTTP server stopped                      endpoint=[::]:8545
INFO [07-25|19:31:00.673] HTTP server stopped                      endpoint=[::]:8546
INFO [07-25|19:31:00.673] IPC endpoint closed                      url=/mnt/ftmmain/node/opera.ipc
INFO [07-25|19:31:00.673] Stopping Fantom protocol
[2024-07-25 19:31:00] [node_command.sh] receive the service exit signal
[2024-07-25 19:31:00] [node_command.sh] get service pid [52958] by command [/opt/ftmmain/core/sonicd]
[2024-07-25 19:31:00] [node_command.sh] exec command [kill -15 52958], try count [1]
WARN [07-25|19:31:00.689] Already shutting down, interrupt more to panic. times=9
INFO [07-25|19:31:01.878] New block                                index=86112135 id=294520:366:a6c24d gas_used=4,049,643 txs=4/0 age=5d23h5m t=1.616s
INFO [07-25|19:31:01.879] Fantom protocol stopped
INFO [07-25|19:31:01.970] New block                                index=86112136 id=294520:384:e278cd gas_used=1,613,396 txs=2/0 age=5d23h5m t=92.498ms
INFO [07-25|19:31:01.990] Fantom service stopped
INFO [07-25|19:31:01.991] Closing State DB...                      module=evm-store
[2024-07-25 19:31:10] [node_command.sh] get service pid [52958] by command [/opt/ftmmain/core/sonicd]
[2024-07-25 19:31:10] [node_command.sh] exec command [kill -15 52958], try count [2]
WARN [07-25|19:31:10.705] Already shutting down, interrupt more to panic. times=8
[2024-07-25 19:31:20] [node_command.sh] get service pid [52958] by command [/opt/ftmmain/core/sonicd]
[2024-07-25 19:31:20] [node_command.sh] exec command [kill -15 52958], try count [3]
WARN [07-25|19:31:20.718] Already shutting down, interrupt more to panic. times=7
[2024-07-25 19:31:30] [node_command.sh] get service pid [52958] by command [/opt/ftmmain/core/sonicd]
[2024-07-25 19:31:30] [node_command.sh] exec command [kill -15 52958], try count [4]
WARN [07-25|19:31:30.736] Already shutting down, interrupt more to panic. times=6
[2024-07-25 19:31:40] [node_command.sh] get service pid [52958] by command [/opt/ftmmain/core/sonicd]
[2024-07-25 19:31:40] [node_command.sh] exec command [kill -15 52958], try count [5]
WARN [07-25|19:31:40.750] Already shutting down, interrupt more to panic. times=5
[2024-07-25 19:31:50] [node_command.sh] get service pid [52958] by command [/opt/ftmmain/core/sonicd]
[2024-07-25 19:31:50] [node_command.sh] exec command [kill -15 52958], try count [6]
WARN [07-25|19:31:50.762] Already shutting down, interrupt more to panic. times=4
[2024-07-25 19:35:37] [node_command.sh] exec command [/opt/ftmmain/core/sonicd --datadir=/mnt/ftmmain/node --cache=8192 --mode=rpc --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.corsdomain=* --http.api=admin,eth,web3,net,ftm,txpool,abft,dag --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=admin,eth,web3,net,ftm,txpool,abft,dag --rpc.gascap=0 --rpc.txfeecap=0 --port=30303]
INFO [07-25|19:35:37.304] Maximum peer count                       total=50
INFO [07-25|19:35:37.304] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
failed to initialize the node: failed to make consensus engine: failed to open existing databases: dirty state: gossip-294520: DE

[thaarok]

You are obviously interrupting Sonic forcefully. The State DB closing may take several minutes, you need to wait until it is done. If you kill the process like above, the database is supposed to be corrupted.

[atlasW]

@YuXiaoCoder 你最后怎么弄的？ 我用容器运行节点也遇到了这个问题。

[jajaislanina]

Had two nodes OOMKill and both are not corrupted. Also trying to figure out how to prevent Kubernetes events causing DB corruption (SIGKILL mainly).

[zfy0701]

same apply to us twice, it's a little too easier to corrupt db