[Node Operator Question] OP-Node Walking back L1Block by hash after Canyon

[dachi-nodies]

Issue Description

After upgrading to Canyon as outlined here: #42

I have been seeing "Walking back L1Block by hash" in OP-Node logs for the past two days or so and no sync progress. I tried to also sync from scratch with the upgraded versions but that is showing an estimate of 14+ days (curious if theres a snapshot available to dl)

Node Logs

t=2023-11-18T06:19:54+0100 lvl=info msg="Walking back L1Block by hash"        curr=0xeab174706c95ba08e34a1febad669c0c1cbbcf3dcb085cf3f490f4d5b9667a29:4691829 next=0xcb19518061ff927c7941f5e07691db1e3e56a8ce44bfe3339d1931b6f0d5f798:4691828 l2block=0xa51af9e9d39b2f8d722ed66ee3b9f2b5fed8bd363753617c3199127601c7fa5e:4078503
t=2023-11-18T06:19:54+0100 lvl=info msg="Walking back L1Block by hash"        curr=0xcb19518061ff927c7941f5e07691db1e3e56a8ce44bfe3339d1931b6f0d5f798:4691828 next=0xf1cac253002e633ce92c7cc56ba8db9badf32cf85a713a39cd685546bdbd64ce:4691827 l2block=0xf9a06514eac17d634a57ff977b9f91693a92668eeed338485004ba93d6de4ef2:4078502
t=2023-11-18T06:19:55+0100 lvl=info msg="Walking back L1Block by hash"        curr=0xf1cac253002e633ce92c7cc56ba8db9badf32cf85a713a39cd685546bdbd64ce:4691827 next=0xdcbafc5ae482f15b3be0134d99a6998460a1970c151a041f00a688b59245fb2b:4691826 l2block=0xf702354f753f2fbc97413842c03ea37143352050f5d034e9ec59f6f83469e6fa:4078496
t=2023-11-18T06:19:57+0100 lvl=info msg="Walking back L1Block by hash"        curr=0xdcbafc5ae482f15b3be0134d99a6998460a1970c151a041f00a688b59245fb2b:4691826 next=0x239a485883570f2f731b62fc42f55f75a8350f89cd4cea483797cf6b8c7c1751:4691825 l2block=0x476ffe762e08ad42dade16719309b04c8f9dfc37d620725c27bc27ab2f5bb024:4078484
t=2023-11-18T06:19:57+0100 lvl=info msg="Walking back L1Block by hash"        curr=0x239a485883570f2f731b62fc42f55f75a8350f89cd4cea483797cf6b8c7c1751:4691825 next=0xf3eae94eabfb68dd671aa2ee656d8d25b909633461dff46a9568a5f394ca6c3b:4691824 l2block=0x5281021449d4b9d1c9e32eed6c171f847e858f5728b28b24d2333ac7bdff907e:4078478
t=2023-11-18T06:19:58+0100 lvl=info msg="Walking back L1Block by hash"        curr=0xf3eae94eabfb68dd671aa2ee656d8d25b909633461dff46a9568a5f394ca6c3b:4691824 next=0x8d9fb21dd101d39ab71d437ad62e1967bb036718e02615639db7e982c3280f15:4691823 l2block=0x5b2ea8d42a038b08657d9e537d06370c1d1272f4873673a88e0988175687469a:4078472
t=2023-11-18T06:19:59+0100 lvl=info msg="disconnected from peer"              peer=16Uiu2HAm84oqbjtQwmhnzD3Q2WQfNHutshnpiKQFHrNPEX4tun34 addr=/ip4/65.108.142.23/tcp/9222
t=2023-11-18T06:19:59+0100 lvl=info msg="Walking back L1Block by hash"        curr=0x8d9fb21dd101d39ab71d437ad62e1967bb036718e02615639db7e982c3280f15:4691823 next=0x93dd75ff1424dbacb57e95d6e946d50276573e1fd1107eb93891eea0ef5ce7d0:4691822 l2block=0xb14645b8808e398f889c58ef0bb7e3a4c44a7f09cd912b52adc545a827922719:4078466
t=2023-11-18T06:19:59+0100 lvl=info msg="Walking back L1Block by hash"        curr=0x93dd75ff1424dbacb57e95d6e946d50276573e1fd1107eb93891eea0ef5ce7d0:4691822 next=0x655033d30ae2426297305adc685f8dd528f122d129d32c7d312a5029792168fa:4691821 l2block=0x3674bf11691737070fcc172fd8424f75dfcba6a4e290cb9f81152f822f2ec8ad:4078460
t=2023-11-18T06:20:00+0100 lvl=info msg="Walking back L1Block by hash"        curr=0x655033d30ae2426297305adc685f8dd528f122d129d32c7d312a5029792168fa:4691821 next=0xd41a6a0a2acc859deff82b79e54082f7cc046c840a5da11daee90fbbddccbb6d:4691820 l2block=0x2633b3c5238ecba2194efa1ef3fc9c598d6af7251477674e1b16c8839a90dc7d:4078454
t=2023-11-18T06:20:00+0100 lvl=info msg="Walking back L1Block by hash"        curr=0xd41a6a0a2acc859deff82b79e54082f7cc046c840a5da11daee90fbbddccbb6d:4691820 next=0x2d93f6f51c627bb66c41f4b5707f3494587f530f0ee3e40847e57f73d2db6443:4691819 l2block=0x87a6d6cd0f060f3bddae3c43a9f994ba428dbda71f2aa3d5dd98c81f628d1e23:4078448

[opfocus]

Yes, it does seem like it's taking too long. Can you describe the process that the node has gone through since it started running? I'm curious why the walkback is taking such an extended period, and it appears to be getting close to returning to the initial starting point. Just to confirm, this is an op-goerli node, correct?

[dachi-nodies]

I have it set to a sepolia eth node.

…

On Sun, Nov 19, 2023 at 6:58 AM simple8720 ***@***.***> wrote: Yes, it does seem like it's taking too long. Can you describe the process that the node has gone through since it started running? I'm curious why the walkback is taking such an extended period, and it appears to be getting close to returning to the initial starting point. Just to confirm, this is an op-goerli node, correct? — Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCNVNGMQVE7U4DFZPOTHZ3TYFH67VAVCNFSM6AAAAAA7QXDMRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMMJRGAYDQ> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[opfocus]

How is it going now

[0xtrou]

This is just normal behavior, the node needs to sync with the whole network, and it should take time

[sbvegan]

To provide some additional context, the node might be walking its way all the way back to genesis. To diagnose this problem further you can:

1. stop the op-node
2. keep op-geth running
3. query the safe, unsafe, and finalized blocks to see where op-geth is at

The RPC requests to do (3) are:

Last Finalized block (output from running below command):

curl $RPC_URL \
-X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["finalized", false],"id":0}'

Latest block (output from running below command):

curl $RPC_URL \
-X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":0}'

Last Safe block (output from running below command):

curl $RPC_URL \
-X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["safe", false],"id":0}'

If there is a large difference between unsafe and safe, that’ll explain why its walking back. If safe and finalize are unavailable, that means they have to walk to genesis to regenerate. If its one of these cases, we can try to put together a script to set the set the unsafe, safe, and finalized block in the database. Or you can clear your database and start fresh with removedb

[sbvegan]

cc @dachi-nodies let me know when you get a chance to try this

[sbvegan]

I'm going to close this due to inactivity. If you provide more information, we can revisit.