Missing some block hash -> block number links in DB after restart | "block body not found" error #30119

ngotchac · 2024-07-04T21:33:10Z

System information

On Linux w/ prysm as the beacon node.

After upgrading from v1.13.15 to v1.14.5 (the node uptime was 30 days), there seem to be some missing block hash to block number links.

The issue appeared by running a simple eth_getLogs query, which returned the error: "block body not found".

After some digging, I found out that this was caused by some missing block hash to block number links:

$> seq 20233380 1 20233399 | xargs -I% -P32 bash -c 'cast logs -j --from-block % --to-block % >/dev/null || echo %'
Error:
server returned an error response: error code -32000: block body not found
20233388
Error:
server returned an error response: error code -32000: block body not found
20233389
Error:
server returned an error response: error code -32000: block body not found
20233390

This failed as well:

cast block 0x173d6a9ce10e1571cd5c72b4d7fadcf2764fcbfaf6836e78e393216ab17da20b

while

cast block 20233390

works.

I wrote this simple geth db subcommand:

func checkMissingBlocks(ctx *cli.Context) error {
	stack, cfg := makeConfigNode(ctx)
	defer stack.Close()

	backend, err := eth.New(stack, &cfg.Eth)
	if err != nil {
		return err
	}

	var (
		latestBlock = backend.BlockChain().HeaderChain().CurrentHeader().Number.Uint64()
		startTime   = time.Now()
		lastLog     = time.Now()
		limiter     = make(chan struct{}, 128)
		lock        sync.Mutex
		wg          sync.WaitGroup
	)
	log.Info("Iterating the blocks", "latest", fmt.Sprintf("%d", latestBlock))
	for bn := uint64(1); bn < latestBlock; bn++ {
		bn := bn
		limiter <- struct{}{}
		wg.Add(1)
		go func() {
			defer func() {
				<-limiter
				wg.Done()
			}()

			h := rawdb.ReadCanonicalHash(backend.ChainDb(), bn)
			if h == (common.Hash{}) {
				log.Error(fmt.Sprintf("block number %d w/ no hash", bn))
				return
			}

			dbBN := rawdb.ReadHeaderNumber(backend.ChainDb(), h)
			if dbBN == nil || *dbBN != bn {
				log.Error(fmt.Sprintf("failed to get body: number=%d: %v", bn, dbBN))
				return
			}

			lock.Lock()
			if time.Since(lastLog) > 2*time.Second {
				log.Info("Iterating the blocks", "at", fmt.Sprintf("%d", bn), "elapsed", common.PrettyDuration(time.Since(startTime)))
				lastLog = time.Now()
			}
			lock.Unlock()
		}()
	}
	wg.Wait()
	return nil
}

to check for missing links, and the 3 block numbers were reported as problematic.

We can see in the logs:

Rewound to block with state number=20,233,387 after restart, which is the last valid block post-restart before the 3 missing links
The Latest filled block is not available error

Logs

2024-07-04 13:41:48.485	[info] Chain head was updated                   number=20,233,516 hash=3f9d34..11a1a9 root=c0c7d6..afb64d elapsed=1.531143ms
2024-07-04 13:42:00.452	[info] Imported new potential chain segment     number=20,233,517 hash=2e5944..6872f6 blocks=1 txs=127  mgas=9.479  elapsed=35.151ms    mgasps=269.666  snapdiffs=6.02MiB  triediffs=213.49MiB triedirty=63.85MiB
2024-07-04 13:42:00.525	[info] Chain head was updated                   number=20,233,517 hash=2e5944..6872f6 root=3e886d..5a6ea8 elapsed=1.230687ms
2024-07-04 13:42:14.789	[info] Imported new potential chain segment     number=20,233,518 hash=2d209c..0f2855 blocks=1 txs=308  mgas=22.577 elapsed=470.881ms   mgasps=47.947   snapdiffs=6.08MiB  triediffs=214.38MiB triedirty=0.00B
2024-07-04 13:42:14.828	[info] Chain head was updated                   number=20,233,518 hash=2d209c..0f2855 root=84ad92..003d91 elapsed=2.576692ms
2024-07-04 13:42:25.344	[info] Imported new potential chain segment     number=20,233,519 hash=ad38a3..0974be blocks=1 txs=131  mgas=11.587 elapsed=42.520ms    mgasps=272.499  snapdiffs=6.12MiB  triediffs=213.31MiB triedirty=2.10MiB
2024-07-04 13:42:25.396	[info] Chain head was updated                   number=20,233,519 hash=ad38a3..0974be root=19427f..15023a elapsed=3.442031ms
2024-07-04 13:42:36.481	[info] Imported new potential chain segment     number=20,233,520 hash=efec36..f2e44f blocks=1 txs=170  mgas=12.440 elapsed=45.501ms    mgasps=273.395  snapdiffs=6.16MiB  triediffs=213.47MiB triedirty=2.96MiB
2024-07-04 13:42:36.531	[info] Chain head was updated                   number=20,233,520 hash=efec36..f2e44f root=370672..75343e elapsed=1.565638ms
2024-07-04 13:42:49.606	[info] Imported new potential chain segment     number=20,233,521 hash=0e6861..0641a9 blocks=1 txs=172  mgas=16.520 elapsed=58.737ms    mgasps=281.250  snapdiffs=6.21MiB  triediffs=213.32MiB triedirty=4.20MiB
2024-07-04 13:42:49.702	[info] Chain head was updated                   number=20,233,521 hash=0e6861..0641a9 root=c88f7f..646dd0 elapsed=1.773487ms
2024-07-04 13:43:00.291	[info] Imported new potential chain segment     number=20,233,522 hash=51b06e..a410e8 blocks=1 txs=176  mgas=15.866 elapsed=52.017ms    mgasps=305.021  snapdiffs=6.26MiB  triediffs=213.44MiB triedirty=5.06MiB
2024-07-04 13:43:00.339	[info] Chain head was updated                   number=20,233,522 hash=51b06e..a410e8 root=d8db28..3bcbd3 elapsed=1.550817ms
2024-07-04 13:43:12.509	[info] Imported new potential chain segment     number=20,233,523 hash=c52a37..7c615b blocks=1 txs=146  mgas=12.829 elapsed=53.395ms    mgasps=240.273  snapdiffs=6.30MiB  triediffs=212.89MiB triedirty=6.25MiB
2024-07-04 13:43:12.548	[info] Chain head was updated                   number=20,233,523 hash=c52a37..7c615b root=4fb230..f865fc elapsed=1.344081ms
2024-07-04 13:43:21.870	[info] Got interrupt, shutting down...
2024-07-04 13:43:21.870	[info] HTTP server stopped                      endpoint=127.0.0.1:8551
2024-07-04 13:43:21.870	[info] HTTP server stopped                      endpoint=[::]:8545
2024-07-04 13:43:21.870	[info] IPC endpoint closed                      url=/var/lib/geth/geth.ipc
2024-07-04 13:43:21.871	[info] Ethereum protocol stopped
2024-07-04 13:43:21.871	[info] Transaction pool stopped
2024-07-04 13:43:21.893	[info] Persisting dirty state to disk           head=20,233,523 root=4fb230..f865fc layers=133
2024-07-04 13:43:22.555	[info] Persisted dirty state to disk            size=204.65MiB elapsed=662.786ms
2024-07-04 13:43:22.556	[info] Blockchain stopped
2024-07-04 13:43:56.871	[info] Looking for peers                        peercount=1 tried=66 static=7
2024-07-04 13:44:31.872	[info] Looking for peers                        peercount=1 tried=7  static=7
2024-07-04 13:45:06.873	[info] Looking for peers                        peercount=1 tried=7  static=7
2024-07-04 13:45:14.922	[warn] Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!
2024-07-04 13:45:41.873	[info] Looking for peers                        peercount=1 tried=7  static=7
2024-07-04 13:46:16.874	[info] Looking for peers                        peercount=1 tried=7  static=7
2024-07-04 13:46:51.875	[info] Looking for peers                        peercount=1 tried=7  static=7
2024-07-04 13:47:26.875	[info] Looking for peers                        peercount=1 tried=7  static=7
2024-07-04 13:48:01.875	[info] Looking for peers                        peercount=1 tried=7  static=7
2024-07-04 13:48:22.017	[info] Enabling metrics collection
2024-07-04 13:48:22.017	[info] Enabling stand-alone metrics HTTP endpoint address=0.0.0.0:9101
2024-07-04 13:48:22.017	[info] Starting Geth on Ethereum mainnet...     version=1.14.5
2024-07-04 13:48:22.017	[info] Starting metrics server                  addr=http://0.0.0.0:9101/debug/metrics
2024-07-04 13:48:22.019	[info] Maximum peer count                       ETH=50 total=50
2024-07-04 13:48:22.020	[info] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
2024-07-04 13:48:22.023	[info] Set global gas cap                       cap=50,000,000
2024-07-04 13:48:22.024	[info] Initializing the KZG library             backend=gokzg
2024-07-04 13:48:22.047	[info] Allocated trie memory caches             clean=1.20GiB dirty=2.00GiB
2024-07-04 13:48:22.144	[info] Allocated cache and file handles         database=/var/lib/geth/geth/chaindata cache=4.00GiB handles=262,144
2024-07-04 13:48:22.144	[info] Using pebble as the backing database
2024-07-04 13:48:26.761	[info] Opened ancient database                  database=/var/lib/geth/geth/chaindata/ancient/chain readonly=false
2024-07-04 13:48:26.762	[info] State scheme set by user                 scheme=path
2024-07-04 13:48:26.764	[info] Initialising Ethereum protocol           network=1 dbversion=8
2024-07-04 13:48:26.764	[warn] Sanitizing invalid node buffer size      provided=2.00GiB updated=256.00MiB
2024-07-04 13:48:26.862	[info] Failed to load journal, discard it       err="unexpected journal version want 1 got 0"
2024-07-04 13:48:26.926	[info] Opened ancient database                  database=/var/lib/geth/geth/chaindata/ancient/state readonly=false
2024-07-04 13:48:26.928	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=account.data items=1,921,281 limit=1,921,276
2024-07-04 13:48:26.928	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=account.index items=1,921,281 limit=1,921,276
2024-07-04 13:48:26.928	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=history.meta items=1,921,281 limit=1,921,276
2024-07-04 13:48:26.928	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=storage.data items=1,921,281 limit=1,921,276
2024-07-04 13:48:26.928	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=storage.index items=1,921,281 limit=1,921,276
2024-07-04 13:48:26.929	[warn] Truncated extra state histories          number=5
2024-07-04 13:48:40.266	[info]  - Homestead:                   #1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
2024-07-04 13:48:40.266	[info] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2024-07-04 13:48:40.266	[info] Chain ID:  1 (mainnet)
2024-07-04 13:48:40.266	[info] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
2024-07-04 13:48:40.266	[info] Pre-Merge hard forks (block based):
2024-07-04 13:48:40.267	[info]  - Arrow Glacier:               #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
2024-07-04 13:48:40.267	[info]  - Berlin:                      #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
2024-07-04 13:48:40.267	[info]  - Byzantium:                   #4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
2024-07-04 13:48:40.267	[info]  - Cancun:                      @1710338135 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
2024-07-04 13:48:40.267	[info]  - Constantinople:              #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
2024-07-04 13:48:40.267	[info]  - DAO Fork:                    #1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
2024-07-04 13:48:40.267	[info]  - Gray Glacier:                #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
2024-07-04 13:48:40.267	[info]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
2024-07-04 13:48:40.267	[info]  - Istanbul:                    #9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
2024-07-04 13:48:40.267	[info]  - London:                      #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
2024-07-04 13:48:40.267	[info]  - Muir Glacier:                #9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
2024-07-04 13:48:40.267	[info]  - Network known to be merged: true
2024-07-04 13:48:40.267	[info]  - Petersburg:                  #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
2024-07-04 13:48:40.267	[info]  - Shanghai:                    @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
2024-07-04 13:48:40.267	[info]  - Spurious Dragon/1 (EIP 155): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2024-07-04 13:48:40.267	[info]  - Spurious Dragon/2 (EIP 158): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2024-07-04 13:48:40.267	[info]  - Tangerine Whistle (EIP 150): #2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
2024-07-04 13:48:40.267	[info]  - Total terminal difficulty:  58750000000000000000000
2024-07-04 13:48:40.267	[info] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2024-07-04 13:48:40.267	[info] Merge configured:
2024-07-04 13:48:40.267	[info] Post-Merge hard forks (timestamp based):
2024-07-04 13:48:40.267	[info] INFO [07-04|13:48:40.266]
2024-07-04 13:48:40.267	[info] INFO [07-04|13:48:40.267]
2024-07-04 13:48:40.268	[info] Loaded last snap-sync pivot marker       number=18,312,114
2024-07-04 13:48:40.268	[info] Loaded most recent local block           number=20,233,523 hash=c52a37..7c615b td=58,750,003,716,598,352,816,469 age=5m29s
2024-07-04 13:48:40.268	[info] Loaded most recent local finalized block number=20,233,458 hash=b890de..18ed48 td=58,750,003,716,598,352,816,469 age=18m41s
2024-07-04 13:48:40.338	[warn] Head state missing, repairing            number=20,233,523 hash=c52a37..7c615b snaproot=6d9fcf..b17454
2024-07-04 13:48:40.578	[info] Rewound to block with state              number=20,233,387 hash=16daf0..92f5e6
2024-07-04 13:48:40.578	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=account.data items=1,921,276 limit=1,921,273
2024-07-04 13:48:40.578	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=account.index items=1,921,276 limit=1,921,273
2024-07-04 13:48:40.578	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=history.meta items=1,921,276 limit=1,921,273
2024-07-04 13:48:40.578	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=storage.data items=1,921,276 limit=1,921,273
2024-07-04 13:48:40.578	[warn] Truncating freezer table                 database=/var/lib/geth/geth/chaindata/ancient/state table=storage.index items=1,921,276 limit=1,921,273
2024-07-04 13:48:51.171	[info] Loaded last snap-sync pivot marker       number=18,312,114
2024-07-04 13:48:51.171	[info] Loaded most recent local block           number=20,233,387 hash=16daf0..92f5e6 td=58,750,003,716,598,352,816,469 age=33m4s
2024-07-04 13:48:51.171	[info] Loaded most recent local finalized block number=20,233,458 hash=b890de..18ed48 td=58,750,003,716,598,352,816,469 age=18m52s
2024-07-04 13:48:51.171	[info] Loaded most recent local header          number=20,233,523 hash=c52a37..7c615b td=58,750,003,716,598,352,816,469 age=5m40s
2024-07-04 13:48:51.171	[info] Loaded most recent local snap block      number=20,233,523 hash=c52a37..7c615b td=58,750,003,716,598,352,816,469 age=5m40s
2024-07-04 13:48:51.171	[warn] Enabling snapshot recovery               chainhead=20,233,387 diskbase=20,233,387
2024-07-04 13:48:51.347	[info] Initialized transaction indexer          range="entire chain"
2024-07-04 13:48:51.347	[warn] Snapshot is not continuous with chain    snaproot=4fb230..f865fc chainroot=6d9fcf..b17454
2024-07-04 13:48:51.348	[info] Loaded local transaction journal         transactions=1 dropped=0
2024-07-04 13:48:51.348	[info] Regenerated local transaction journal    transactions=1 accounts=1
2024-07-04 13:48:51.348	[info] Setting new local account                address=0xcafeb939C55Aa7aFCa1bbbE6987685FD4e306d9E
2024-07-04 13:48:51.382	[warn] Dropping dangling blob transactions      from=0x5050F69a9786F081509234F1a7F4684b5E5b76C9 missing=449,630 drop=[449655] ids=[1342177280]
2024-07-04 13:48:51.383	[warn] Dropping dangling blob transactions      from=0x000000633b68f5D8D3a86593ebB815b4663BCBe0 missing=130,731 drop=[130789] ids=[268435456]
2024-07-04 13:48:51.412	[info] Gasprice oracle is ignoring threshold set threshold=2
2024-07-04 13:48:51.412	[warn] Switch sync mode from snap sync to full sync reason="snap sync complete"
2024-07-04 13:48:51.413	[info] Starting peer-to-peer node               instance=Geth/v1.14.5-stable-7328d001/linux-amd64/go1.22.1
2024-07-04 13:48:51.413	[warn] Engine API enabled                       protocol=eth
2024-07-04 13:48:51.413	[warn] Unclean shutdown detected                booted=2023-10-14T12:29:06+0000 age=8mo3w3d
2024-07-04 13:48:51.529	[info] New local node record                    seq=1,690,985,487,727 id=4087202d4e8885b8 ip=127.0.0.1 udp=30303 tcp=30303
2024-07-04 13:48:51.529	[info] Started P2P networking                   self=enode://40fa3a4561d6ddbc7ed653199e7316e3ef98947ef98bda9c16b4cc5001710ff49cbb93eba4f212f288a86efce5833f00ba086c6d24b6125c5c97c0aef7bd6e50@127.0.0.1:30303
2024-07-04 13:48:51.530	[info] IPC endpoint opened                      url=/var/lib/geth/geth.ipc
2024-07-04 13:48:51.531	[info] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=localhost
2024-07-04 13:48:51.531	[info] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors= vhosts=*
2024-07-04 13:48:51.531	[info] Loaded JWT secret file                   path=/var/lib/geth/geth/jwtsecret crc32=0x8d873ad
2024-07-04 13:48:51.531	[info] WebSocket enabled                        url=ws://127.0.0.1:8551
2024-07-04 13:48:51.531	[info] WebSocket enabled                        url=ws://[::]:8545
2024-07-04 13:48:52.033	[info] > Connecting to /var/lib/geth/geth.ipc...
2024-07-04 13:48:52.033	[info] > IPC socket at /var/lib/geth/geth.ipc doesn't exist. Waiting...
2024-07-04 13:48:52.033	[info] > Sending a `eth_blockNumber` request...
2024-07-04 13:48:52.034	[info]   Node at block 20233387
2024-07-04 13:48:52.034	[info] > Successfully connected to /var/lib/geth/geth.ipc
2024-07-04 13:48:54.456	[info] New local node record                    seq=1,690,985,487,728 id=4087202d4e8885b8 ip=57.128.92.15 udp=30303 tcp=30303
2024-07-04 13:48:55.347	[warn] State not available, ignoring new payload
2024-07-04 13:48:55.453	[info] Forkchoice requested sync to new head    number=20,233,524 hash=59690f..e94a85 finalized=unknown
2024-07-04 13:48:55.870	[error] Latest filled block is not available
2024-07-04 13:48:55.870	[info] Block synchronisation started
2024-07-04 13:48:55.870	[info] Syncing beacon headers                   downloaded=1 left=0 eta=0s
2024-07-04 13:48:55.871	[warn] Retrieved pivot header from local        number=20,233,460 hash=88ed6b..96ba60 latest=20,233,524 oldest=20,233,523
2024-07-04 13:49:01.311	[info] Importing sidechain segment              start=20,233,388 end=20,233,523
2024-07-04 13:49:01.736	[info] Forkchoice requested sync to new head    number=20,233,525 hash=cde0a3..d91a19 finalized=unknown
2024-07-04 13:49:02.801	[info] Forkchoice requested sync to new head    number=20,233,526 hash=bf4483..4ce072 finalized=unknown
2024-07-04 13:49:04.649	[info] Forkchoice requested sync to new head    number=20,233,527 hash=9f3246..c8f605 finalized=unknown
2024-07-04 13:49:06.872	[info] Forkchoice requested sync to new head    number=20,233,528 hash=24ab27..0b715f finalized=unknown
2024-07-04 13:49:07.840	[info] Forkchoice requested sync to new head    number=20,233,529 hash=b75db3..bfe7af finalized=unknown
2024-07-04 13:49:09.354	[info] Imported new chain segment               number=20,233,463 hash=24b18d..2372ea blocks=76 txs=13396 mgas=1215.636 elapsed=8.042s mgasps=151.145 age=18m10s  snapdiffs=6.26MiB triediffs=129.08MiB triedirty=0.00B
2024-07-04 13:49:10.554	[info] Forkchoice requested sync to new head    number=20,233,530 hash=7bab48..ce5abd finalized=unknown
2024-07-04 13:49:11.952	[info] Forkchoice requested sync to new head    number=20,233,531 hash=77bdc6..7777c5 finalized=unknown
2024-07-04 13:49:13.794	[info] Forkchoice requested sync to new head    number=20,233,532 hash=7bb642..e724a3 finalized=unknown
2024-07-04 13:49:14.559	[info] Imported new chain segment               number=20,233,523 hash=c52a37..7c615b blocks=60 txs=10688 mgas=894.878  elapsed=5.204s mgasps=171.930 age=6m3s    snapdiffs=6.25MiB triediffs=211.37MiB triedirty=8.65MiB
2024-07-04 13:49:14.734	[info] Imported new chain segment               number=20,233,524 hash=59690f..e94a85 blocks=1  txs=225   mgas=19.206   elapsed=174.429ms mgasps=110.107 age=5m51s   snapdiffs=6.31MiB triediffs=212.37MiB triedirty=9.12MiB
2024-07-04 13:49:14.921	[info] Forkchoice requested sync to new head    number=20,233,533 hash=077a3b..290e92 finalized=unknown
2024-07-04 13:49:14.922	[warn] Retrieved pivot header from local        number=20,233,469 hash=0cdfeb..39a7cf latest=20,233,533 oldest=20,233,523
2024-07-04 13:49:16.599	[info] Imported new chain segment               number=20,233,533 hash=077a3b..290e92 blocks=9  txs=1820  mgas=125.469  elapsed=1.251s    mgasps=100.224 age=4m5s    snapdiffs=6.69MiB triediffs=210.99MiB triedirty=15.59MiB ignored=1
2024-07-04 13:49:20.165	[info] Imported new potential chain segment     number=20,233,534 hash=34ef0f..fd2c66 blocks=1  txs=434   mgas=26.055   elapsed=169.019ms mgasps=154.156 age=3m57s   snapdiffs=6.77MiB triediffs=212.69MiB triedirty=16.10MiB
2024-07-04 13:49:20.627	[info] Chain head was updated                   number=20,233,534 hash=34ef0f..fd2c66 root=997372..3564db elapsed=2.852867ms age=3m57s
2024-07-04 13:49:20.797	[info] Imported new potential chain segment     number=20,233,535 hash=0ba004..25c9d8 blocks=1  txs=119   mgas=9.066    elapsed=120.319ms  mgasps=75.350  age=3m45s   snapdiffs=6.80MiB triediffs=212.08MiB triedirty=16.72MiB
2024-07-04 13:49:21.625	[info] Chain head was updated                   number=20,233,535 hash=0ba004..25c9d8 root=cfe891..fca26e elapsed=2.271597ms age=3m46s
2024-07-04 13:49:21.720	[info] Imported new potential chain segment     number=20,233,536 hash=92caeb..3811da blocks=1  txs=26    mgas=2.421    elapsed=50.773ms   mgasps=47.682  age=3m34s   snapdiffs=6.81MiB triediffs=210.55MiB triedirty=17.35MiB

The text was updated successfully, but these errors were encountered:

protolambda · 2024-08-21T22:50:46Z

The "failed to load journal" log may be related to a journal-format change in #28940

I have implemented a possible compatibility improvement in our op-geth fork here: ethereum-optimism/op-geth#368

ngotchac added the type:bug label Jul 4, 2024

sebastianst mentioned this issue Aug 21, 2024

Database corruption during restarts since switching to upstream v1.14.7 ethereum-optimism/op-geth#360

Closed

fjl added the status:triage label Aug 29, 2024

gballet assigned rjl493456442 Sep 12, 2024

gballet removed the status:triage label Sep 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing some block hash -> block number links in DB after restart | "block body not found" error #30119

Missing some block hash -> block number links in DB after restart | "block body not found" error #30119

ngotchac commented Jul 4, 2024 •

edited

Loading

protolambda commented Aug 21, 2024

Missing some block hash -> block number links in DB after restart | "block body not found" error #30119

Missing some block hash -> block number links in DB after restart | "block body not found" error #30119

Comments

ngotchac commented Jul 4, 2024 • edited Loading

System information

Logs

protolambda commented Aug 21, 2024

ngotchac commented Jul 4, 2024 •

edited

Loading