Follower Node Block Hash Mismatch with Testnet at Specific Heights

[wengych]

Describe the bug
I configured a follower node using the documentation provided by Movement: Follower Node Documentation. The corresponding .env configuration is as follows:

CONTAINER_REV=a4ba7a656f429e79bbb637898b451c21159a3dc0
DOT_MOVEMENT_PATH=./.movement
MAPTOS_CHAIN_ID=250 # change this to the chain id of the network you are running
MOVEMENT_SYNC="follower::mtnet-l-sync-bucket-sync<=>{maptos,maptos-storage,suzuka-da-db}/**" # change to the sync bucket for the network you are running
M1_DA_LIGHT_NODE_CONNECTION_PROTOCOL=https
M1_DA_LIGHT_NODE_CONNECTION_HOSTNAME="m1-da-light-node.testnet.movementnetwork.xyz"
M1_DA_LIGHT_NODE_CONNECTION_PORT=443
AWS_ACCESS_KEY_ID=xxxx
AWS_SECRET_ACCESS_KEY=xxxxxx
AWS_DEFAULT_REGION=us-west-1 # change this to match the region of the sync bucket
AWS_REGION=us-west-1 # change this to match the region of the sync bucket

The command to start the node is as follows:
sudo just suzuka-full-node docker-compose follower

When I ran it for the first time using just, the API returned the following data:

curl http://localhost:30731/v1
{"chain_id":250,"epoch":"1","ledger_version":"51","oldest_ledger_version":"0","ledger_timestamp":"1728914733620485","node_role":"validator","oldest_block_height":"0","block_height":"18","git_hash":"70be3926ff79ff4cdb0cee928f717fafcd41ecdd"}

After cross-referencing the block height with the testnet, I found that the hash for the same block height does not match:

• Localhost: curl http://localhost:30731/v1/blocks/by_height/18

  {"block_height":"18","block_hash":"0x8a12c83f0146c1a13d4be394d7071f4775c8c99dc8cffb61225780b5ade0c4ef","block_timestamp":"1728914733620485","first_version":"50","last_version":"51","transactions":null}
  

• Testnet: curl https://testnet.movementnetwork.xyz/v1/blocks/by_height/18

  {"block_height":"18","block_hash":"0xc0e1930bca19784d53a633b6b96845a3f0b99852a482caf24e9db30cbf5ac647","block_timestamp":"1728770354144339","first_version":"51","last_version":"53","transactions":null}
  

I then cleared the data in .movement and restarted the node. The API now returns updated progress:

curl http://localhost:30731/v1/
{"chain_id":250,"epoch":"1","ledger_version":"11494","oldest_ledger_version":"0","ledger_timestamp":"1728958250399194","node_role":"validator","oldest_block_height":"0","block_height":"3820","git_hash":"70be3926ff79ff4cdb0cee928f717fafcd41ecdd"}

Comparing this with the testnet data:

curl https://testnet.movementnetwork.xyz/v1
{"chain_id":250,"epoch":"1","ledger_version":"11969","oldest_ledger_version":"0","ledger_timestamp":"1728965454500874","node_role":"validator","oldest_block_height":"0","block_height":"3978","git_hash":"70be3926ff79ff4cdb0cee928f717fafcd41ecdd"}

Based on the returned block height 3820, I queried the last three blocks:

• Localhost:
  • curl http://localhost:30731/v1/blocks/by_height/3820

    {"block_height":"3820","block_hash":"0x3a7e580ff2bed2d3eda8b25a35e58fc8f59a45bdd7ee4e3b83d220c2c88e9f50","block_timestamp":"1728958250399194","first_version":"11493","last_version":"11494","transactions":null}
    

  • curl http://localhost:30731/v1/blocks/by_height/3819

    {"block_height":"3819","block_hash":"0x4d8b89947616653ce13be6b240c08015b00aa5761021bea6d3729df6ab3b2515","block_timestamp":"1728947883012323","first_version":"11491","last_version":"11492","transactions":null}
    

  • curl http://localhost:30731/v1/blocks/by_height/3818

    {"block_height":"3818","block_hash":"0xac4ef81afa40c657180fba2e0a1b908f03357a1bf8c2fb3583c3205cab64eb6d","block_timestamp":"1728947725906075","first_version":"11488","last_version":"11490","transactions":null}
    

• Testnet:
  • curl https://testnet.movementnetwork.xyz/v1/blocks/by_height/3820

    {"block_height":"3820","block_hash":"0xa885a780f80e28f1baa48069e282b20d848b9b4dd1003ee21496fe6196a4263b","block_timestamp":"1728950369666229","first_version":"11493","last_version":"11495","transactions":null}
    

  • curl https://testnet.movementnetwork.xyz/v1/blocks/by_height/3819

    {"block_height":"3819","block_hash":"0xf7cdad579375b08ae7550f05e34760acb272bcef5c7b4f155c4702640d77cd1c","block_timestamp":"1728950357309721","first_version":"11491","last_version":"11492","transactions":null}
    

  • curl https://testnet.movementnetwork.xyz/v1/blocks/by_height/3818

    {"block_height":"3818","block_hash":"0xac4ef81afa40c657180fba2e0a1b908f03357a1bf8c2fb3583c3205cab64eb6d","block_timestamp":"1728947725906075","first_version":"11488","last_version":"11490","transactions":null}
    

Blocks 3820 and 3819 are inconsistent between the local node and the testnet.

To Reproduce
Steps to reproduce the behavior:

1. Follow the documentation to configure a follower node: Follower Node Documentation
2. Use the provided .env configuration.
3. Start the follower node with the command: sudo just suzuka-full-node docker-compose follower.
4. Query block height data and compare it with the testnet.

Expected behavior
The block data should match the corresponding blocks from the testnet to ensure consistency.

Screenshots

Desktop (please complete the following information):

• macOS 15
• Server: NixOS 24.05.5630.c505ebf77752 (Uakari)

Additional context
The follower node appears to be out of sync with the testnet despite restarting and clearing local data. Block hashes and heights are inconsistent.

[l-monninger]

This may be related to replays vs. syncing under #689, but I will see if I can reproduce and investigate.

[zfy0701]

any updates on that?

[l-monninger]

I believe I've figured it out. I likely won't have a fix in until next week.

[wengych]

The issue still seems to persist. When running a follower node locally, after a period of time, the ledger_version and block_height remain stuck at 8795 and 2564, respectively. Meanwhile, the results queried through https://aptos.testnet.bardock.movementlabs.xyz/v1 show continuous growth. Additionally, the local node_role is set as validator instead of follower. Could this also be a problem?

[wengych]

Same as #860