LIB stale after switching to Savanna

[ericpassmore]

After activating Savanna algorithm http://host:port/v1/chain/get_info and cleos get info report a stale value for last_irreversible_block_num. The logs show LIB advancing.

Follow bootstrap-private-network for a reproducible case
https://github.com/eosnetworkfoundation/bootstrap-private-network/blob/9d8cae4/README.md

Using leap build from the following commit 49c96a20392993c291cfd03b731759c6c6ca2bef
https://github.com/eosnetworkfoundation/bootstrap-private-network/blob/9d8cae4dfd54481028fda652d7ec8c4c44bec891/AntelopeDocker#L46

[ericpassmore]

Ah, my mistake LIB isn't advancing at all. Investigating why.

[ericpassmore]

info  2024-03-04T20:18:48.897 nodeos    controller.cpp:2829           operator()           ] Transition to instant finality happening after block 170
info  2024-03-04T20:18:48.899 nodeos    producer_plugin.cpp:743       on_incoming_block    ] Received block b502d176b9e0ca17... #170 @ 2024-03-04T20:18:49.000 signed by bpc [trxs: 1, lib: 170, net: 1392, cpu: 4657, elapsed: 8895, time: 10700, latency: -100 ms]
info  2024-03-04T20:18:49.351 nodeos    producer_plugin.cpp:743       on_incoming_block    ] Received block 928f6867f2159815... #171 @ 2024-03-04T20:18:49.500 signed by bpc [trxs: 0, lib: 170, net: 0, cpu: 100, elapsed: 69, time: 197, latency: -148 ms]
info  2024-03-04T20:18:49.813 nodeos    producer_plugin.cpp:743       on_incoming_block    ] Received block d9a2a2e2b9ad26c9... #172 @ 2024-03-04T20:18:50.000 signed by bpc [trxs: 0, lib: 170, net: 0, cpu: 100, elapsed: 50, time: 158, latency: -186 ms]
info  2024-03-04T20:18:50.276 nodeos    producer_plugin.cpp:743       on_incoming_block    ] Received block 83702ef9d34f2001... #173 @ 2024-03-04T20:18:50.500 signed by bpc [trxs: 0, lib: 170, net: 0, cpu: 100, elapsed: 104, time: 220, latency: -223 ms]
info  2024-03-04T20:18:50.738 nodeos    producer_plugin.cpp:743       on_incoming_block    ] Received block af4170fd563f7454... #174 @ 2024-03-04T20:18:51.000 signed by bpc [trxs: 0, lib: 170, net: 0, cpu: 100, elapsed: 45, time: 153, latency: -261 ms]
info  2024-03-04T20:18:51.201 nodeos    producer_plugin.cpp:743       on_incoming_block    ] Received block 0be1abbd504a43f5... #175 @ 2024-03-04T20:18:51.500 signed by bpc [trxs: 0, lib: 170, net: 0, cpu: 100, elapsed: 59, time: 177, latency: -298 ms]

[ericpassmore]

grep final /bigata1/log/nodeos-*.log
/bigata1/log/nodeos-one.log:info  2024-03-04T20:18:48.897 nodeos    controller.cpp:2829           operator()           ] Transition to instant finality happening after block 170
/bigata1/log/nodeos-three.log:info  2024-03-04T20:18:48.888 nodeos    controller.cpp:2829           operator()           ] Transition to instant finality happening after block 170
/bigata1/log/nodeos-two.log:info  2024-03-04T20:18:48.897 nodeos    controller.cpp:2829           operator()           ] Transition to instant finality happening after block 170

[ericpassmore]

User error not a real issue. Needed to write the BLS finalizer keys to the configuration, then restart all the instances to re-load the configuraiton, and then run setfinalizer.