Cherry pick bc4b3fcd449f3376f9e891245d9fb31e11016c13 into aptos-release-v1.9

[banool]

Context: https://aptos-org.slack.com/archives/C06BBLE5AHK.

[trunk-io]

⏱️ 4m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
run-tests-main-branch	4m	🟩
file_change_determinator	13s	🟩
permission-check	3s	🟩
permission-check	3s	🟩
permission-check	2s	🟩
permission-check	2s	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[sherry-x]

I see it’s being manually tested, but can we add some unit test case there to cover the functionality? I don’t expect we’re using this feature frequently in production, and we need to make sure it works when we need it. Also we'd need to document how to use it so that during emergency people can actually apply the config change.

[banool]

PR to add tests: #11796.

[bchocho]

LGTM

[github-actions]

✅ Forge suite realistic_env_max_load success on eb3cc2b94007de5948ed6f9774fc9bd6583e72d3

two traffics test: inner traffic : committed: 8596 txn/s, latency: 4533 ms, (p50: 4200 ms, p90: 4800 ms, p99: 12900 ms), latency samples: 3704900
two traffics test : committed: 100 txn/s, latency: 2349 ms, (p50: 2100 ms, p90: 2400 ms, p99: 13800 ms), latency samples: 1740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.255, avg: 0.210", "QsPosToProposal: max: 0.120, avg: 0.117", "ConsensusProposalToOrdered: max: 0.563, avg: 0.518", "ConsensusOrderedToCommit: max: 0.439, avg: 0.417", "ConsensusProposalToCommit: max: 0.985, avg: 0.935"]
Max round gap was 1 [limit 4] at version 1751294. Max no progress secs was 9.890124 [limit 10] at version 1751294.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on aptos-node-v1.8.3 ==> eb3cc2b94007de5948ed6f9774fc9bd6583e72d3

Compatibility test results for aptos-node-v1.8.3 ==> eb3cc2b94007de5948ed6f9774fc9bd6583e72d3 (PR)
1. Check liveness of validators at old version: aptos-node-v1.8.3
compatibility::simple-validator-upgrade::liveness-check : committed: 4533 txn/s, latency: 6644 ms, (p50: 6600 ms, p90: 10800 ms, p99: 13600 ms), latency samples: 181320
2. Upgrading first Validator to new version: eb3cc2b94007de5948ed6f9774fc9bd6583e72d3
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1770 txn/s, latency: 16426 ms, (p50: 19000 ms, p90: 22300 ms, p99: 22400 ms), latency samples: 92060
3. Upgrading rest of first batch to new version: eb3cc2b94007de5948ed6f9774fc9bd6583e72d3
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1783 txn/s, latency: 16119 ms, (p50: 19300 ms, p90: 22000 ms, p99: 22500 ms), latency samples: 92740
4. upgrading second batch to new version: eb3cc2b94007de5948ed6f9774fc9bd6583e72d3
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3209 txn/s, latency: 9292 ms, (p50: 9900 ms, p90: 13600 ms, p99: 14200 ms), latency samples: 138000
5. check swarm health
Compatibility test for aptos-node-v1.8.3 ==> eb3cc2b94007de5948ed6f9774fc9bd6583e72d3 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions]

❌ Forge suite framework_upgrade failure on aptos-node-v1.8.3 ==> eb3cc2b94007de5948ed6f9774fc9bd6583e72d3

Compatibility test results for aptos-node-v1.8.3 ==> eb3cc2b94007de5948ed6f9774fc9bd6583e72d3 (PR)
Upgrade the nodes to version: eb3cc2b94007de5948ed6f9774fc9bd6583e72d3
Test Failed: API error: Unknown error error sending request for url (http://aptos-node-3-validator.forge-framework-upgrade-pr-11714.svc:8080/v1/accounts/0000000000000000000000000000000000000000000000000000000000000001/resource/0x1::block::BlockResource): error trying to connect: dns error: failed to lookup address information: Name or service not known

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: __libc_start_main
  14: <unknown>
Trailing Log Lines:
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: __libc_start_main
  14: <unknown>


Swarm logs can be found here: See fgi output for more information.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ApiError: namespaces "forge-framework-upgrade-pr-11714" not found: NotFound (ErrorResponse { status: "Failure", message: "namespaces \"forge-framework-upgrade-pr-11714\" not found", reason: "NotFound", code: 404 })

Caused by:
    namespaces "forge-framework-upgrade-pr-11714" not found: NotFound

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: __libc_start_main
  16: <unknown>', testsuite/forge/src/backend/k8s/swarm.rs:676:18
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking