Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Failure: read_only_trxs tests's max-transaction-time too small #1816

Closed
linh2931 opened this issue Oct 24, 2023 · 0 comments · Fixed by #1832
Closed

Test Failure: read_only_trxs tests's max-transaction-time too small #1816

linh2931 opened this issue Oct 24, 2023 · 0 comments · Fixed by #1832
Assignees
Labels
👍 lgtm test-instability tag issues for flaky tests, high priority to address

Comments

@linh2931
Copy link
Member

max-transaction-time in read_only_trxs tests was originally set to 10000us. That's before activate_protocol_features_set_bios_contract was added. activate_protocol_features_set_bios_contract calls set_code to set bios contracts which can take more than 10000us; this causes the tests to be stuck.

This issue is split from #1815 so that increasing max-transaction-time can be merged in before the root cause for stuck set_code is figured out.

The following log is from #1815 for reference

...
1989: info  2023-10-24T15:06:49.059 plugin_te producer_plugin.cpp:1242      plugin_initialize    ] read-only-write-window-time-us: 100000 us, read-only-read-window-time-us: 40000 us, effective read window time to be used: 30000 us
1989: info  2023-10-24T15:06:49.059 plugin_te producer_plugin.cpp:1260      plugin_initialize    ] Read-only max transaction time 10000us set to fit in the effective read-only window 30000us.
1989: info  2023-10-24T15:06:49.059 plugin_te producer_plugin.cpp:1262      plugin_initialize    ] read-only-threads 3, max read-only trx time to be enforced: 10000 us
1989: warn  2023-10-24T15:06:49.060 plugin_te controller.cpp:637            startup              ] No existing chain state or fork database. Initializing fresh blockchain state and resetting fork database.
1989: warn  2023-10-24T15:06:49.060 plugin_te controller.cpp:483            initialize_blockchai ] Initializing new blockchain with genesis state
1989: info  2023-10-24T15:06:49.083 plugin_te controller.cpp:557            replay               ] no irreversible blocks need to be replayed
1989: info  2023-10-24T15:06:49.083 plugin_te controller.cpp:573            replay               ] 0 reversible blocks replayed
1989: info  2023-10-24T15:06:49.083 plugin_te controller.cpp:581            replay               ] replayed 0 blocks in 0 seconds, 0.00000000001443550 ms/block
1989: info  2023-10-24T15:06:49.083 plugin_te chain_plugin.cpp:1143         plugin_startup       ] starting chain in read/write mode
1989: info  2023-10-24T15:06:49.083 plugin_te chain_plugin.cpp:1147         plugin_startup       ] Blockchain started; head block is #1, genesis timestamp is 2018-06-01T12:00:00.000
1989: info  2023-10-24T15:06:49.083 plugin_te producer_plugin.cpp:1313      plugin_startup       ] producer plugin:  plugin_startup() begin
1989: info  2023-10-24T15:06:49.083 plugin_te producer_plugin.cpp:1354      plugin_startup       ] Launching block production for 1 producers at 2023-10-24T15:06:49.083.
1989: debug 2023-10-24T15:06:49.084 plugin_te producer_plugin.cpp:1858      start_block          ] Starting block #2 at 2023-10-24T15:06:49.084 producer eosio
1989: debug 2023-10-24T15:06:49.084 plugin_te controller.cpp:2503           clear_expired_input_ ] removed 0 expired transactions of the 0 input dedup list, pending block time 2023-10-24T15:06:49.500
1989: debug 2023-10-24T15:06:49.084 plugin_te producer_plugin.cpp:2015      remove_expired_trxs  ] Processed 0 expired transactions of the 0 transactions in the unapplied queue.
1989: debug 2023-10-24T15:06:49.084 plugin_te producer_plugin.cpp:2523      schedule_maybe_produ ] Scheduling Block Production on Normal Block #2 for 2023-10-24T15:06:49.350
1989: info  2023-10-24T15:06:49.084 plugin_te producer_plugin.cpp:1380      plugin_startup       ] producer plugin:  plugin_startup() end
1989: debug 2023-10-24T15:06:49.350 plugin_te producer_plugin.cpp:2540      operator()           ] Produce block timer for 2 running at 2023-10-24T15:06:49.350
1989: debug 2023-10-24T15:06:49.352 plugin_te producer_plugin.cpp:2582      operator()           ] Signing took 1592us
1989: info  2023-10-24T15:06:49.353 plugin_te producer_plugin.cpp:2642      produce_block        ] Produced block f383cf772570abdb... #2 @ 2023-10-24T15:06:49.500 signed by eosio [trxs: 0, lib: 1, confirmed: 0, net: 0, cpu: 100, elapsed: 139, time: 2942]
1989: debug 2023-10-24T15:06:49.353 plugin_te producer_plugin.cpp:288       report               ] Block #2 eosio trx idle: 265696us out of 269296us, success: 0, 0us, fail: 0, 0us, transient: 0, 0us, other: 3599us
1989: debug 2023-10-24T15:06:49.353 plugin_te producer_plugin.cpp:1858      start_block          ] Starting block #3 at 2023-10-24T15:06:49.353 producer eosio
1989: debug 2023-10-24T15:06:49.355 plugin_te controller.cpp:2503           clear_expired_input_ ] removed 0 expired transactions of the 0 input dedup list, pending block time 2023-10-24T15:06:50.000
1989: debug 2023-10-24T15:06:49.355 plugin_te producer_plugin.cpp:2015      remove_expired_trxs  ] Processed 0 expired transactions of the 0 transactions in the unapplied queue.
1989: debug 2023-10-24T15:06:49.355 plugin_te producer_plugin.cpp:2523      schedule_maybe_produ ] Scheduling Block Production on Normal Block #3 for 2023-10-24T15:06:49.812
1989: debug 2023-10-24T15:06:49.355 plugin_te producer_plugin.cpp:2542      operator()           ] Producing Block #2 returned: true
1989: debug 2023-10-24T15:06:49.660 plugin_te producer_plugin.cpp:2077      log_trx_results      ] [TRX_TRACE] Block 3 for producer eosio is REJECTING tx: 0170c14d176ba71424c5c18aaaf7c913c9e470d1acd892409147386ffb6da497, auth: eosio, action: eosio:setcode, transaction 0170c14d176ba71424c5c18aaaf7c913c9e470d1acd892409147386ffb6da497 was executing for too long 10843us reached node configured max-transaction-time 10000us
1989: debug 2023-10-24T15:06:49.662 plugin_te producer_plugin.cpp:2084      log_trx_results      ] [TRX_TRACE] Block 3 for producer eosio is REJECTING tx: {"expiration":"2023-10-24T15:07:19","ref_block_num":2,"ref_block_prefix":3685445669,"max_net_usage_words":0,"max_cpu_usage_ms":0,"delay_sec":0,"context_free_actions":[],"actions":[{"account":"eosio","name":"setcode","authorization":[{"actor":"eosio","permission":"active"}],"code_hash":"241108ec5076fe0df9dbe9e6aab249bc26cef0362f6805ff43118ab74115cabd","data":{"account":"eosio","vmtype":0,"vmversion":0,"code":{"size":19646,"trimmed_hex":"0061736d010000000198011960000060027f7f0060037f7f7f0060047e7e7e7e017f6000017e60047f7e7e7f0060057f7f7f7f7f017f60037f7f7f017f60027f"}},"hex_data":{"size":19659,"trimmed_hex":"0000000000ea30550000be99010061736d010000000198011960000060027f7f0060037f7f7f0060047e7e7e7e017f6000017e60047f7e7e7f0060057f7f7f7f"}}]}
1989: debug 2023-10-24T15:06:49.664 plugin_te producer_plugin.cpp:2087      log_trx_results      ] [TRX_TRACE] Block 3 for producer eosio is REJECTING tx: {"id":"0170c14d176ba71424c5c18aaaf7c913c9e470d1acd892409147386ffb6da497","block_num":3,"block_time":"2023-10-24T15:06:50.000","producer_block_id":null,"receipt":null,"elapsed":19731,"net_usage":6874,"scheduled":false,"action_traces":[{"action_ordinal":1,"creator_action_ordinal":0,"closest_unnotified_ancestor_action_ordinal":0,"receipt":null,"receiver":"eosio","act":{"account":"eosio","name":"setcode","authorization":[{"actor":"eosio","permission":"active"}],"code_hash":"241108ec5076fe0df9dbe9e6aab249bc26cef0362f6805ff43118ab74115cabd","data":{"account":"eosio","vmtype":0,"vmversion":0,"code":{"size":19646,"trimmed_hex":"0061736d010000000198011960000060027f7f0060037f7f7f0060047e7e7e7e017f6000017e60047f7e7e7f0060057f7f7f7f7f017f60037f7f7f017f60027f"}},"hex_data":{"size":19659,"trimmed_hex":"0000000000ea30550000be99010061736d010000000198011960000060027f7f0060037f7f7f0060047e7e7e7e017f6000017e60047f7e7e7f0060057f7f7f7f"}},"context_free":false,"elapsed":18788,"console":"","trx_id":"0170c14d176ba71424c5c18aaaf7c913c9e470d1acd892409147386ffb6da497","block_num":3,"block_time":"2023-10-24T15:06:50.000","producer_block_id":null,"account_ram_deltas":[{"account":"eosio","delta":196460}],"except":{"code":3080004,"name":"tx_cpu_usage_exceeded","message":"Transaction exceeded the current CPU usage limit imposed on the transaction","stack":[{"context":{"level":"error","file":"transaction_context.cpp","line":478,"method":"checktime","hostname":"","thread_name":"plugin_test","timestamp":"2023-10-24T15:06:49.660"},"format":"transaction ${id} was executing for too long ${billing_timer}us reached node configured max-transaction-time ${limit}us","data":{"id":"0170c14d176ba71424c5c18aaaf7c913c9e470d1acd892409147386ffb6da497","billing_timer":10843,"subjective":0,"limit":10000}},{"context":{"level":"warn","file":"apply_context.cpp","line":134,"method":"exec_one","hostname":"","thread_name":"plugin_test","timestamp":"2023-10-24T15:06:49.660"},"format":"pending console output: ${console}","data":{"console":""}}]},"error_code":"10000000000000000000","return_value_hex_data":""}],"account_ram_delta":null,"except":{"code":3080004,"name":"tx_cpu_usage_exceeded","message":"Transaction exceeded the current CPU usage limit imposed on the transaction","stack":[{"context":{"level":"error","file":"transaction_context.cpp","line":478,"method":"checktime","hostname":"","thread_name":"plugin_test","timestamp":"2023-10-24T15:06:49.660"},"format":"transaction ${id} was executing for too long ${billing_timer}us reached node configured max-transaction-time ${limit}us","data":{"id":"0170c14d176ba71424c5c18aaaf7c913c9e470d1acd892409147386ffb6da497","billing_timer":10843,"subjective":0,"limit":10000}},{"context":{"level":"warn","file":"apply_context.cpp","line":134,"method":"exec_one","hostname":"","thread_name":"plugin_test","timestamp":"2023-10-24T15:06:49.660"},"format":"pending console output: ${console}","data":{"console":""}}]},"error_code":"10000000000000000000"}
1989: debug 2023-10-24T15:06:49.813 plugin_te producer_plugin.cpp:2540      operator()           ] Produce block timer for 3 running at 2023-10-24T15:06:49.813
1989: debug 2023-10-24T15:06:49.814 plugin_te producer_plugin.cpp:2582      operator()           ] Signing took 387us

...

1989: info  2023-10-24T15:31:40.590 plugin_te producer_plugin.cpp:2642      produce_block        ] Produced block 596350d1ff5fc553... #2985 @ 2023-10-24T15:31:41.000 signed by eosio [trxs: 0, lib: 2984, confirmed: 0, net: 0, cpu: 100, elapsed: 279, time: 2681]
1989: debug 2023-10-24T15:31:40.590 plugin_te producer_plugin.cpp:288       report               ] Block #2985 eosio trx idle: 458364us out of 462042us, success: 0, 0us, fail: 0, 0us, transient: 0, 0us, other: 3677us
1989: debug 2023-10-24T15:31:40.590 plugin_te producer_plugin.cpp:1858      start_block          ] Starting block #2986 at 2023-10-24T15:31:40.590 producer eosio
1989: debug 2023-10-24T15:31:40.591 plugin_te controller.cpp:2503           clear_expired_input_ ] removed 0 expired transactions of the 0 input dedup list, pending block time 2023-10-24T15:31:41.500
1989: debug 2023-10-24T15:31:40.591 plugin_te producer_plugin.cpp:2015      remove_expired_trxs  ] Processed 0 expired transactions of the 0 transactions in the unapplied queue.
1989: debug 2023-10-24T15:31:40.591 plugin_te producer_plugin.cpp:2523      schedule_maybe_produ ] Scheduling Block Production on Normal Block #2986 for 2023-10-24T15:31:41.050
1989: debug 2023-10-24T15:31:40.592 plugin_te producer_plugin.cpp:2542      operator()           ] Producing Block #2985 returned: true
1/1 Test #1989: plugin_test ......................***Timeout 1500.02 sec
@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog Oct 24, 2023
@linh2931 linh2931 added the test-instability tag issues for flaky tests, high priority to address label Oct 24, 2023
@linh2931 linh2931 self-assigned this Oct 24, 2023
@linh2931 linh2931 moved this from Todo to In Progress in Team Backlog Oct 24, 2023
@linh2931 linh2931 moved this from In Progress to Awaiting Review in Team Backlog Oct 24, 2023
@bhazzard bhazzard removed the triage label Oct 26, 2023
@bhazzard bhazzard added this to the Leap v5.0.1 milestone Oct 26, 2023
@github-project-automation github-project-automation bot moved this from Awaiting Review to Done in Team Backlog Oct 26, 2023
@bhazzard bhazzard modified the milestones: Leap v5.0.1, Leap v5.0.0-rc3 Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
👍 lgtm test-instability tag issues for flaky tests, high priority to address
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants