Snapshot fails to load when forkdb and shared_memory are removed

[ericpassmore]

The Error message is
Error disabling address space randomization
Thread 1 "nodeos" received signal SIG34, Real-time event 34.

Leap built from commit 118a401deb2699662fd87adb47e26b4a7bfa41e1
Using config and settings from https://github.com/eosnetworkfoundation/bootstrap-private-network/tree/ehp/March12

[ericpassmore]

Starting program: /usr/local/bin/nodeos --agent-name Finality\ Test\ Node\ Two --http-server-address 0.0.0.0:6888 --p2p-listen-endpoint 0.0.0.0:2444 --producer-name bpb --signature-provider EOS7bY84sefAiVp9rpoG449rDtz2JJgrziJ81MU5fS7oyXEf6JQdU=KEY:5KRQWpVuCRPipdTrdhm7baiRVJsmJ5CKyBdYhpkJXK8mS1faXFb --config /bigata1/savanna/config.ini --data-dir /bigata1/savanna/nodeos-two/data --p2p-peer-address 127.0.0.1:1444 --p2p-peer-address 127.0.0.1:3444 --snapshot /bigata1/savanna/nodeos-one/data/snapshots/snapshot-00000775d10342da22ecdac32f1bb96f582d8e585774db8eb26b4e8f245f4b93.bin
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 1117]
info  2024-03-12T21:53:32.268 nodeos    main.cpp:151                  main                 ] nodeos started
[New Thread 0x7f1e17ed7640 (LWP 1119)]
APPBASE: Warning: The following configuration items in the config.ini file are redundantly set to
         their default value:
             allowed-connection
         Explicit values will override future changes to application defaults. Consider commenting out or
         removing these items.

[ericpassmore]

info  2024-03-12T21:53:32.269 nodeos    chain_plugin.cpp:490          plugin_initialize    ] initializing chain plugin
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'DISABLE_DEFERRED_TRXS_STAGE_1' (with digest of 'fce57d2331667353a0eac6b4209b67b843a7262a848af0a49a6e2fa9f6584eb4') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'DISABLE_DEFERRED_TRXS_STAGE_2' (with digest of '09e86cb0accf8d81c9e85d34bea4b925ae936626d00c984e4691186891f5bc16') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:888    operator()           ] Support for builtin protocol feature 'PREACTIVATE_FEATURE' (with digest of '0ec7e080177b2c02b278d5088611686b49d739925a92d9bfcacd7fc6b74053bd') is enabled without activation restrictions
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'ONLY_LINK_TO_EXISTING_PERMISSION' (with digest of '1a99a59d87e06e09ec5b028a9cbb7749b4a5ad8819004365d02dc4379a8b7241') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'FORWARD_SETCODE' (with digest of '2652f5f96006294109b3dd0bbde63693f55324af452b799ee137a81a905eed25') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'WTMSIG_BLOCK_SIGNATURES' (with digest of '299dcb6af692324b899b39f16d5a530a33062804e41f09dc97e9f156b4476707') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'GET_BLOCK_NUM' (with digest of '35c2186cc36f7bb4aeaf4487b36e57039ccf45a9136aa856a5d569ecca55ef2b') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'REPLACE_DEFERRED' (with digest of 'ef43112c6543b88db2283a2e077278c315ae2c84719a8b25f25cc88565fbea99') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'NO_DUPLICATE_DEFERRED_ID' (with digest of '4a90c00d55454dc5b059055ca213579c6ea856967712a56017487886a4d4cc0f') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'RAM_RESTRICTIONS' (with digest of '4e7bf348da00a945489b2a681749eb56f5de00b900014e137ddae39f48f69d67') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'WEBAUTHN_KEY' (with digest of '4fca8bd82bbd181e714e283f83e1b45d95ca5af40fb89ad3977b653c448f78c2') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'BLOCKCHAIN_PARAMETERS' (with digest of '5443fcf88330c586bc0e5f3dee10e7f63c76c00249c87fe4fbf7f38c082006b4') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'BLS_PRIMITIVES2' (with digest of '63320dd4a58212e4d32d1f58926b73ca33a247326c2a5e9fd39268d2384e011a') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'DISALLOW_EMPTY_PRODUCER_SCHEDULE' (with digest of '68dcaa34c0517d19666e6b33add67351d8c5f69e999ca1e37931bc410a297428') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'CRYPTO_PRIMITIVES' (with digest of '6bcb40a24e49c26d0a60513b6aeb8551d264e4717f306b81a37a5afb3b47cedc') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'ONLY_BILL_FIRST_AUTHORIZER' (with digest of '8ba52fe7a3956c5cd3a656a3174b931d3bb2abb45578befc59f283ecd816a405') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'INSTANT_FINALITY' (with digest of '8cb6dd1e5607208331eb5983141e159c75a597413887e80e8a9a4b715a507eb7') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'RESTRICT_ACTION_TO_SELF' (with digest of 'ad9e3d8f650687709fd68f4b90b41f7d825a365b02c23a636cef88ac2ac00c43') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'GET_CODE_HASH' (with digest of 'bcd2a26394b36614fd4894241d3c451ab0f6fd110958c3423073621a70826e99') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'ACTION_RETURN_VALUE' (with digest of 'c3a6138c5061cf291310887c0b5c71fcaffeab90d5deb50d3b9e687cead45071') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'CONFIGURABLE_WASM_LIMITS2' (with digest of 'd528b9f6e9693f45ed277af93474fd473ce7d831dae2180cca35d907bd10cb40') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'FIX_LINKAUTH_RESTRICTION' (with digest of 'e0fb64b1085cc5538970158d05a009c24e276fb94e1a0bf6a528b48fbc4ff526') is enabled with preactivation required
info  2024-03-12T21:53:32.270 nodeos    protocol_feature_manag:875    operator()           ] Support for builtin protocol feature 'GET_SENDER' (with digest of 'f0af56d2c5a48d60a4a5b5c903edfb7db3a736a94ed589d0b797df33ff9d3e1d') is enabled with preactivation required
info  2024-03-12T21:53:32.278 nodeos    block_log.cpp:670             open                 ] Log has 1957 blocks
[New Thread 0x7f1e0ae8f640 (LWP 1120)]
[New Thread 0x7f1e0a68e640 (LWP 1121)]
[New Thread 0x7f1e09e8d640 (LWP 1122)]
info  2024-03-12T21:53:32.289 nodeos    producer_plugin.cpp:1209      plugin_initialize    ] Subjective CPU billing disabled
info  2024-03-12T21:53:32.289 nodeos    producer_plugin.cpp:1291      plugin_initialize    ] Read-only max transaction time 0us set to fit in the effective read-only window 0us.
info  2024-03-12T21:53:32.289 nodeos    producer_plugin.cpp:1293      plugin_initialize    ] read-only-threads 0, max read-only trx time to be enforced: 0 us
warn  2024-03-12T21:53:32.289 nodeos    producer_api_plugin.cp:149    plugin_initialize    ] 
**********SECURITY WARNING**********
*                                  *
* --       Producer RW API      -- *
* - EXPOSED to the LOCAL NETWORK - *
* - USE ONLY ON SECURE NETWORKS! - *
*                                  *
************************************

warn  2024-03-12T21:53:32.289 nodeos    producer_api_plugin.cp:160    plugin_initialize    ] 
**********SECURITY WARNING**********
*                                  *
* --         Snapshot API       -- *
* - EXPOSED to the LOCAL NETWORK - *
* - USE ONLY ON SECURE NETWORKS! - *
*                                  *
************************************

info  2024-03-12T21:53:32.289 nodeos    net_plugin.cpp:4238           plugin_initialize    ] Initialize net plugin
warn  2024-03-12T21:53:32.289 nodeos    net_api_plugin.cpp:73         plugin_initialize    ] 
**********SECURITY WARNING**********
*                                  *
* --        Net RW API          -- *
* - EXPOSED to the LOCAL NETWORK - *
* - USE ONLY ON SECURE NETWORKS! - *
*                                  *
************************************

info  2024-03-12T21:53:32.289 nodeos    resource_monitor_plugi:70     plugin_initialize    ] Monitoring interval set to 2
info  2024-03-12T21:53:32.289 nodeos    resource_monitor_plugi:86     plugin_initialize    ] Space usage threshold set to 90%
info  2024-03-12T21:53:32.289 nodeos    resource_monitor_plugi:96     plugin_initialize    ] Shutdown flag when threshold exceeded set to true
info  2024-03-12T21:53:32.289 nodeos    resource_monitor_plugi:103    plugin_initialize    ] Warning interval set to 30
info  2024-03-12T21:53:32.290 nodeos    main.cpp:190                  main                 ] nodeos version v5.1.0-dev v5.1.0-dev-118a401deb2699662fd87adb47e26b4a7bfa41e1
info  2024-03-12T21:53:32.290 nodeos    main.cpp:193                  main                 ] nodeos using configuration file /bigata1/savanna/config.ini
info  2024-03-12T21:53:32.290 nodeos    main.cpp:194                  main                 ] nodeos data directory is /bigata1/savanna/nodeos-two/data
info  2024-03-12T21:53:32.290 nodeos    main.cpp:62                   log_non_default_opti ] Non-default options: agent-name = Finality Test Node Two, http-server-address = 0.0.0.0:6888, p2p-listen-endpoint = 0.0.0.0:2444, producer-name = bpb, signature-provider = ***, config = /bigata1/savanna/config.ini, data-dir = /bigata1/savanna/nodeos-two/data, p2p-peer-address = 127.0.0.1:1444, p2p-peer-address = 127.0.0.1:3444, snapshot = /bigata1/savanna/nodeos-one/data/snapshots/snapshot-00000775d10342da22ecdac32f1bb96f582d8e585774db8eb26b4e8f245f4b93.bin, eos-vm-oc-enable = 1, chain-state-db-size-mb = 200, verbose-http-errors = true, max-transaction-time = 100000, allowed-connection = any, plugin = eosio::chain_api_plugin, plugin = eosio::chain_plugin, plugin = eosio::http_plugin, plugin = eosio::producer_plugin, plugin = eosio::producer_api_plugin, plugin = eosio::net_plugin, plugin = eosio::net_api_plugin, plugin = eosio::db_size_api_plugin, disable-replay-opts = true, read-only-read-window-time-us = 165000, read-only-write-window-time-us = 50000, signature-provider = ***, signature-provider = ***, signature-provider = ***
[New Thread 0x7f1e0968c640 (LWP 1123)]
info  2024-03-12T21:53:32.290 nodeos    controller.cpp:1490           startup              ] Starting initialization from snapshot and block log 1-1957, this may take a significant amount of time
info  2024-03-12T21:53:32.302 nodeos    controller.cpp:1501           startup              ] Snapshot loaded, lib: 1909
info  2024-03-12T21:53:32.302 nodeos    controller.cpp:1354           replay_block_log     ] existing block log, attempting to replay from 1910 to 1957 blocks

[ericpassmore]

Thread 1 "nodeos" received signal SIG34, Real-time event 34.
0x00005646f7e8a84d in eosio::chain::compute_and_print_timer_accuracy(eosio::chain::platform_timer&) ()
(gdb) bt
#0  0x00005646f7e8a84d in eosio::chain::compute_and_print_timer_accuracy(eosio::chain::platform_timer&) ()
#1  0x00005646f7d7afb0 in eosio::chain::platform_timer::platform_timer() ()
#2  0x00005646f745f5c9 in __tls_init.part.0 ()
#3  0x00005646f76152b5 in eosio::chain::controller_impl::push_transaction(std::shared_ptr<eosio::chain::transaction_metadata> const&, fc::time_point, fc::microseconds, unsigned int, bool, long) ()
#4  0x00005646f7781f42 in eosio::chain::controller_impl::start_block(eosio::chain::block_timestamp<(unsigned short)500, 946684800000ul>, unsigned short, std::vector<fc::sha256, std::allocator<fc::sha256> > const&, eosio::chain::controller::block_status, std::optional<fc::sha256> const&, fc::time_point const&) ()
#5  0x00005646f77a8c16 in void eosio::chain::controller_impl::apply_block<std::shared_ptr<eosio::chain::block_state> >(eosio::chain::controller::block_report&, std::shared_ptr<eosio::chain::block_state> const&, eosio::chain::controller::block_status, std::function<std::shared_ptr<eosio::chain::transaction_metadata> (fc::sha256 const&)> const&) ()
#6  0x00005646f77be038 in auto eosio::chain::controller_impl::replay_push_block<std::shared_ptr<eosio::chain::block_state> >(std::shared_ptr<eosio::chain::signed_block> const&, eosio::chain::controller::block_status)::{lambda(auto:1 const&)#2}::operator()<std::shared_ptr<eosio::chain::block_state> >(std::shared_ptr<eosio::chain::block_state> const&) const ()
#7  0x00005646f74b2866 in void eosio::chain::controller_impl::replay_push_block<std::shared_ptr<eosio::chain::block_state> >(std::shared_ptr<eosio::chain::signed_block> const&, eosio::chain::controller::block_status) ()
#8  0x00005646f778dbcb in eosio::chain::controller_impl::replay_block_log(std::function<bool ()> const&) ()
#9  0x00005646f778f6b9 in eosio::chain::controller_impl::replay(std::function<bool ()> const&, eosio::chain::controller_impl::startup_t) ()
#10 0x00005646f77c30b5 in eosio::chain::controller_impl::init(std::function<bool ()>, eosio::chain::controller_impl::startup_t) ()
#11 0x00005646f77cec91 in eosio::chain::controller_impl::startup(std::function<void ()>, std::function<bool ()>, std::shared_ptr<eosio::chain::snapshot_reader> const&) ()
#12 0x00005646f7471e1f in eosio::chain::controller::startup(std::function<void ()>, std::function<bool ()>, std::shared_ptr<eosio::chain::snapshot_reader> const&) ()
#13 0x00005646f70813b0 in eosio::chain_plugin_impl::plugin_startup() ()
#14 0x00005646f6bfebe6 in appbase::application_base::startup(boost::asio::io_context&) ()
#15 0x00005646f6b75a2e in main ()

[spoonincode]

Thread 1 "nodeos" received signal SIG34, Real-time event 34.

SIG34 is a real time signal used for the deadline timer. Usually you want to disable gdb from stopping on it: handle SIG34 noprint

[greg7mdp]

Eric, yes there was an issue restarting a leap node from a post IF-transition snapshot because the fork_database issue 2285 was not implemented yet.
It is now fixed in my PR #2306, although it might be safer to do so

If you try with commit 8d4d5a0, it should now work. You probably will not have to recreate the snapshot, although it might be safer to do so.

[ericpassmore]

Verified this works ! Closing