feat!: submit eviction proof for nodes that miss many proposals

[sdbondi]

Description

feat!: submit eviction proof for nodes that miss many proposals
various bug fixes
cucumber that tests end-to-end eviction
watcher submits the eviction transaction
fixes for epoch manager to allow exits
removed suspending and resuming nodes
exit consensus and sync if receiving a QC for a future epoch

Motivation and Context

If a node goes offline, consensus will run much more slowly due to having to wait for leader failure each round.

This PR implements the eviction command and generation of an eviction proof that can be submitted on L1. Once the epoch is reached where the eviction applies, the validator is removed from the committee. Requires tari-project/tari#6702

Suspending and resuming nodes adds complexity and some bugs were observed. Since evicted nodes are already immediately skipped in rounds by other validator nodes before they are evicted, the value of suspending nodes is limited and it was decided that it was not worth the added complexity.

How Has This Been Tested?

New unit tests and cucumber, manually

What process can a PR reviewer use to test or verify this change?

1. Run a swarm,
2. run a watcher and let it register
3. mine 10 blocks to allow the watcher vn to start consensus
4. stop a node in swarm
5. wait for eviction proof to be submitted by the watcher
6. mine 10 blocks, observe that the stopped node is no longer part of the committee

Breaking Changes

• None
• Requires data directory to be deleted
• Other - Please specify

BREAKING CHANGE: incompatible global and state database changes

[github-actions]

Test Results (CI)

564 tests  +3   564 ✅ +3   1h 35m 37s ⏱️ - 26m 31s
 54 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 05f48a3. ± Comparison against base commit 581a285.

This pull request removes 3 and adds 6 tests. Note that renamed tests count towards both.

consensus_tests ‑ consensus::leader_failure_node_goes_down_and_gets_suspended
tari_dan_storage ‑ consensus_models::command::export_bindings_resumenodeatom
tari_dan_storage ‑ consensus_models::command::export_bindings_suspendnodeatom

consensus_tests ‑ consensus::leader_failure_node_goes_down_and_gets_evicted
tari_consensus ‑ hotstuff::eviction_proof::tests::it_produces_a_summarized_header_that_hashes_to_the_original
tari_validator_node_client ‑ types::export_bindings_getbaselayerepochchangesrequest
tari_validator_node_client ‑ types::export_bindings_getbaselayerepochchangesresponse
tari_validator_node_client ‑ types::export_bindings_getconsensusstatusresponse
tari_validator_node_client ‑ types::export_bindings_validatornodechange

♻️ This comment has been updated with latest results.