Various variants of "simplified/dummy" executors #15152

igor-aptos · 2024-11-01T05:31:59Z

Description

Adding all these executor configurations, for understanding upper bounds on different approaches/optimizations:

    /// Transaction execution: AptosVM
    /// Executing conflicts: in the input order, via BlockSTM,
    /// State: BlockSTM-provided MVHashMap-based view with caching
    AptosVMWithBlockSTM,
    /// Transaction execution: NativeVM - a simplified rust implemtation to create VMChangeSet,
    /// Executing conflicts: in the input order, via BlockSTM
    /// State: BlockSTM-provided MVHashMap-based view with caching
    NativeVMWithBlockSTM,
    /// Transaction execution: AptosVM
    /// Executing conflicts: All transactions execute on the state at the beginning of the block
    /// State: Raw CachedStateView
    AptosVMParallelUncoordinated,
    /// Transaction execution: Native rust code producing WriteSet
    /// Executing conflicts: All transactions execute on the state at the beginning of the block
    /// State: Raw CachedStateView
    NativeParallelUncoordinated,
    /// Transaction execution: Native rust code updating in-memory state, no WriteSet output
    /// Executing conflicts: All transactions execute on the state in the first come - first serve basis
    /// State: In-memory DashMap with rust values of state (i.e. StateKey -> Resource (either Account or FungibleStore)),
    ///        cached across blocks, filled upon first request
    NativeValueCacheParallelUncoordinated,
    /// Transaction execution: Native rust code updating in-memory state, no WriteSet output
    /// Executing conflicts: All transactions execute on the state in the first come - first serve basis
    /// State: In-memory DashMap with AccountAddress to seq_num and balance (ignoring all other fields).
    ///        kept across blocks, randomly initialized on first access, storage ignored.
    NativeNoStorageParallelUncoordinated,

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-11-01T05:32:03Z

⏱️ 16h 4m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
single-node-performance	11h 56m	🟥 🟥 🟥 🟥 🟥 (+16 more)
test-target-determinator	1h 17m	🟩 🟩 🟩 🟩 🟩 (+17 more)
rust-cargo-deny	25m	🟩 🟩 🟩 🟩 🟩 (+10 more)
check-dynamic-deps	14m	🟩 🟩 🟩 🟩 🟩 (+10 more)
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟥
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

manudhundi · 2024-11-21T23:01:33Z

execution/executor-benchmark/src/native/native_transaction.rs

+    },
+}
+
+impl NativeTransaction {


In my remote sharding experiments, native vm was not able to handle this one txn in the beginning. You have changed the code since then, but it might worthwhile to run an "unsharded execution" in the executor-benchmark

2024-11-16T00:33:08.305178Z [main] INFO storage/aptosdb/src/state_store/mod.rs:451 Initializing BufferedState. {"latest_snapshot_version":null,"num_transactions":0} 2024-11-16T00:33:08.305256Z [main] INFO storage/aptosdb/src/state_store/mod.rs:547 StateStore initialization finished. {"latest_in_memory_root_hash":"5350415253455f4d45524b4c455f504c414345484f4c4445525f484153480000","latest_in_memory_version":null,"latest_snapshot_root_hash":"5350415253455f4d45524b4c455f504c414345484f4c4445525f484153480000","latest_snapshot_version":null} 2024-11-16T00:33:08.305735Z [main] INFO aptos-move/aptos-vm/src/aptos_vm.rs:2557 Executing block, transaction count: 1 {"name":"miscellaneous","txn_idx":0}

manudhundi · 2024-11-21T23:48:37Z

execution/executor-benchmark/src/native/mod.rs

@@ -1,4 +1,8 @@
 // Copyright (c) Aptos Foundation
 // SPDX-License-Identifier: Apache-2.0

+pub mod aptos_vm_uncoordinated;


Is this the best place to have 'native' module ??
How about having the native module in "aptos-move/aptos-vm/src/block_executor/" ?

Native is used inside of "aptos-move/aptos-vm/src/..." and I suppose it will be used more from 'lower layers'. executor-benchmark seems to be too high level for this. Even in this code, probably need for 'vm_wrapper' comes from native being in executor-benchmark

vm_wrapper is one line, so the tradeoff seems right

we don't want to use it in prod, only for the benchmark. we can create a separate native project if needed later, but this sounds reasonable to me.

putting it in aptos-move/aptos-vm/src/... seems poluting it there.

manudhundi · 2024-11-21T23:59:27Z

execution/executor-benchmark/src/native/parallel_uncoordinated_block_executor.rs

+        ))
+    }
+
+    // fn to_abort(status: TransactionStatus) -> TransactionOutput {


dead code. maybe remove

manudhundi · 2024-11-22T00:00:15Z

execution/executor-benchmark/src/native/parallel_uncoordinated_block_executor.rs

+    total_supply: AtomicU64,
+}
+
+// impl CommonNativeRawTransactionExecutor for NativeNoStorageRawTransactionExecutor {


manudhundi

Left some comments..

But overall, seems good to me

github-actions · 2024-11-23T02:26:08Z

✅ Forge suite `realistic_env_max_load` success on `dee9376e46f0698beb28349c283eb22b169be934`

two traffics test: inner traffic : committed: 14082.96 txn/s, latency: 2824.37 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 4700 ms), latency samples: 5354680
two traffics test : committed: 100.03 txn/s, latency: 2427.88 ms, (p50: 1500 ms, p70: 2000, p90: 2600 ms, p99: 16800 ms), latency samples: 1720
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.275, avg: 1.307", "ConsensusProposalToOrdered: max: 0.325, avg: 0.294", "ConsensusOrderedToCommit: max: 0.377, avg: 0.364", "ConsensusProposalToCommit: max: 0.669, avg: 0.658"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.16s no progress at version 2572668 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 15.51s no progress at version 2572666 (avg 15.51s) [limit 16].
Test Ok

github-actions · 2024-11-23T02:26:37Z

✅ Forge suite `compat` success on `327445f70a3b9474f13a792e8ada9b85809dec86` ==> `dee9376e46f0698beb28349c283eb22b169be934`

Compatibility test results for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> dee9376e46f0698beb28349c283eb22b169be934 (PR)
1. Check liveness of validators at old version: 327445f70a3b9474f13a792e8ada9b85809dec86
compatibility::simple-validator-upgrade::liveness-check : committed: 13794.53 txn/s, latency: 2419.49 ms, (p50: 2100 ms, p70: 2200, p90: 2500 ms, p99: 7700 ms), latency samples: 521020
2. Upgrading first Validator to new version: dee9376e46f0698beb28349c283eb22b169be934
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7224.32 txn/s, latency: 3930.64 ms, (p50: 4500 ms, p70: 4700, p90: 4800 ms, p99: 4900 ms), latency samples: 133260
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6600.96 txn/s, latency: 4622.69 ms, (p50: 4800 ms, p70: 4900, p90: 5000 ms, p99: 6600 ms), latency samples: 249620
3. Upgrading rest of first batch to new version: dee9376e46f0698beb28349c283eb22b169be934
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7767.45 txn/s, latency: 3719.71 ms, (p50: 4200 ms, p70: 4400, p90: 4500 ms, p99: 4600 ms), latency samples: 145420
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6978.09 txn/s, latency: 4376.00 ms, (p50: 4400 ms, p70: 4500, p90: 4600 ms, p99: 6400 ms), latency samples: 262640
4. upgrading second batch to new version: dee9376e46f0698beb28349c283eb22b169be934
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11202.95 txn/s, latency: 2473.59 ms, (p50: 2600 ms, p70: 2800, p90: 3200 ms, p99: 3300 ms), latency samples: 196080
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11169.49 txn/s, latency: 2866.49 ms, (p50: 2700 ms, p70: 3300, p90: 3500 ms, p99: 4200 ms), latency samples: 361620
5. check swarm health
Compatibility test for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> dee9376e46f0698beb28349c283eb22b169be934 passed
Test Ok

github-actions · 2024-11-23T02:26:48Z

✅ Forge suite `framework_upgrade` success on `327445f70a3b9474f13a792e8ada9b85809dec86` ==> `dee9376e46f0698beb28349c283eb22b169be934`

Compatibility test results for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> dee9376e46f0698beb28349c283eb22b169be934 (PR)
Upgrade the nodes to version: dee9376e46f0698beb28349c283eb22b169be934
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1272.51 txn/s, submitted: 1276.04 txn/s, failed submission: 3.54 txn/s, expired: 3.54 txn/s, latency: 2361.67 ms, (p50: 2100 ms, p70: 2400, p90: 3600 ms, p99: 5400 ms), latency samples: 115180
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1290.00 txn/s, submitted: 1292.94 txn/s, failed submission: 2.93 txn/s, expired: 2.93 txn/s, latency: 2317.81 ms, (p50: 2100 ms, p70: 2400, p90: 3600 ms, p99: 4800 ms), latency samples: 114360
5. check swarm health
Compatibility test for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> dee9376e46f0698beb28349c283eb22b169be934 passed
Upgrade the remaining nodes to version: dee9376e46f0698beb28349c283eb22b169be934
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1332.03 txn/s, submitted: 1336.29 txn/s, failed submission: 4.26 txn/s, expired: 4.26 txn/s, latency: 2328.04 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5400 ms), latency samples: 118820
Test Ok

lightmark · 2024-11-28T03:51:20Z

execution/executor-benchmark/src/db_access.rs

        Self::get_value(account_key, state_view)
    }

-    pub fn get_coin_store(
+    pub fn get_fa_store(
+        store_key: &StateKey,


how does this work?
fa store is in the rg.

In db, yes. But that is StateView's implementation detail - StateView handles the translation

igor-aptos requested review from msmouse, grao1991, sitalkedia and zekun000 November 1, 2024 05:31

igor-aptos requested review from lightmark, davidiw, movekevin, wrwg, vgao1996, georgemitenkov and a team as code owners November 1, 2024 05:32

igor-aptos requested a review from gelash as a code owner November 4, 2024 22:34

igor-aptos force-pushed the igor/native_executor_benchmarks branch 8 times, most recently from a9a56e8 to ef4d774 Compare November 5, 2024 21:09

igor-aptos requested review from JoshLind, gregnazario, sasha8, ibalajiarun, danielxiangzl, banool and 0xmaayan as code owners November 5, 2024 21:09

igor-aptos force-pushed the igor/merge_block_and_vm_executor branch 2 times, most recently from 66a9235 to fd3a91a Compare November 5, 2024 21:23

manudhundi reviewed Nov 21, 2024

View reviewed changes

manudhundi reviewed Nov 22, 2024

View reviewed changes

manudhundi approved these changes Nov 22, 2024

View reviewed changes

igor-aptos force-pushed the igor/native_executor_benchmarks branch 5 times, most recently from 4a8b8ed to 8a854ac Compare November 23, 2024 00:37

Various variants of "simplified/dummy" executors

8acda55

igor-aptos force-pushed the igor/native_executor_benchmarks branch from 8a854ac to 4e0e2c9 Compare November 23, 2024 00:37

igor-aptos enabled auto-merge (squash) November 23, 2024 00:38

This comment has been minimized.

Sign in to view

addressing comments

dee9376

igor-aptos force-pushed the igor/native_executor_benchmarks branch from 4e0e2c9 to dee9376 Compare November 23, 2024 01:58

This comment has been minimized.

Sign in to view

igor-aptos merged commit 516f32e into main Nov 23, 2024
48 checks passed

igor-aptos deleted the igor/native_executor_benchmarks branch November 23, 2024 02:56

lightmark reviewed Nov 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various variants of "simplified/dummy" executors #15152

Various variants of "simplified/dummy" executors #15152

igor-aptos commented Nov 1, 2024

trunk-io bot commented Nov 1, 2024 •

edited

Loading

manudhundi Nov 21, 2024

manudhundi Nov 21, 2024

igor-aptos Nov 22, 2024

manudhundi Nov 21, 2024

manudhundi Nov 22, 2024

manudhundi left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 23, 2024

github-actions bot commented Nov 23, 2024

github-actions bot commented Nov 23, 2024

lightmark Nov 28, 2024

igor-aptos Nov 28, 2024

Various variants of "simplified/dummy" executors #15152

Various variants of "simplified/dummy" executors #15152

Conversation

igor-aptos commented Nov 1, 2024

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Nov 1, 2024 • edited Loading

manudhundi Nov 21, 2024

Choose a reason for hiding this comment

manudhundi Nov 21, 2024

Choose a reason for hiding this comment

igor-aptos Nov 22, 2024

Choose a reason for hiding this comment

manudhundi Nov 21, 2024

Choose a reason for hiding this comment

manudhundi Nov 22, 2024

Choose a reason for hiding this comment

manudhundi left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 23, 2024

✅ Forge suite realistic_env_max_load success on dee9376e46f0698beb28349c283eb22b169be934

github-actions bot commented Nov 23, 2024

✅ Forge suite compat success on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> dee9376e46f0698beb28349c283eb22b169be934

github-actions bot commented Nov 23, 2024

✅ Forge suite framework_upgrade success on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> dee9376e46f0698beb28349c283eb22b169be934

lightmark Nov 28, 2024

Choose a reason for hiding this comment

igor-aptos Nov 28, 2024

Choose a reason for hiding this comment

trunk-io bot commented Nov 1, 2024 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `dee9376e46f0698beb28349c283eb22b169be934`

✅ Forge suite `compat` success on `327445f70a3b9474f13a792e8ada9b85809dec86` ==> `dee9376e46f0698beb28349c283eb22b169be934`

✅ Forge suite `framework_upgrade` success on `327445f70a3b9474f13a792e8ada9b85809dec86` ==> `dee9376e46f0698beb28349c283eb22b169be934`