Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Io Gas adjustments #11300

Merged
merged 1 commit into from
Dec 13, 2023
Merged

Io Gas adjustments #11300

merged 1 commit into from
Dec 13, 2023

Conversation

msmouse
Copy link
Contributor

@msmouse msmouse commented Dec 12, 2023

Description

  1. read bytes charged in 4k steps
  2. remove per write op free quota

Test Plan

Copy link

trunk-io bot commented Dec 12, 2023

⏱️ 51h 21m total CI duration on this PR
Job Cumulative Duration Recent Runs
windows-build 5h 28m 🟩🟩🟩🟩🟩 (+15 more)
rust-unit-tests 5h 8m 🟥🟥🟩 (+9 more)
replay-mainnet / replay-verify (16) 4h 9m 🟩
rust-move-unit-coverage 3h 34m 🟩🟩🟩🟩 (+9 more)
replay-mainnet / replay-verify (13) 2h 51m 🟩
replay-mainnet / replay-verify (15) 2h 50m 🟩
rust-images / rust-all 2h 12m 🟩🟩🟩🟩 (+8 more)
replay-mainnet / replay-verify (12) 2h 5m 🟩
replay-mainnet / replay-verify (11) 1h 56m 🟩
replay-mainnet / replay-verify (4) 1h 54m 🟩
rust-lints 1h 37m 🟩🟩🟩🟩 (+9 more)
replay-mainnet / replay-verify (14) 1h 33m 🟩
replay-mainnet / replay-verify (7) 1h 29m 🟩
replay-mainnet / replay-verify (10) 1h 28m 🟩
replay-mainnet / replay-verify (9) 1h 17m 🟩
replay-mainnet / replay-verify (8) 1h 13m 🟩
replay-mainnet / replay-verify (0) 1h 6m 🟩
replay-mainnet / replay-verify (5) 58m 🟩
run-tests-main-branch 55m 🟩🟩🟩🟩 (+9 more)
check 54m 🟩🟩🟩 (+12 more)
replay-mainnet / replay-verify (6) 53m 🟩
replay-mainnet / replay-verify (1) 48m 🟩
replay-mainnet / replay-verify (3) 46m 🟩
forge-framework-upgrade-test / forge 44m 🟥
check-dynamic-deps 43m 🟩🟩🟩🟩🟩 (+15 more)
general-lints 32m 🟩🟩🟩🟩 (+9 more)
rust-smoke-tests 29m 🟩
replay-mainnet / replay-verify (2) 28m 🟩
execution-performance / single-node-performance 18m 🟩
forge-e2e-test / forge 16m 🟩
forge-compat-test / forge 14m 🟩
cli-e2e-tests / run-cli-tests 11m 🟩
semgrep/ci 7m 🟩🟩🟩🟩🟩 (+15 more)
file_change_determinator 3m 🟩🟩🟩🟩🟩 (+9 more)
file_change_determinator 2m 🟩🟩🟩🟩🟩 (+8 more)
file_change_determinator 2m 🟩🟩🟩🟩 (+12 more)
permission-check 58s 🟩🟩🟩🟩🟩 (+15 more)
permission-check 56s 🟩🟩🟩🟩🟩 (+15 more)
node-api-compatibility-tests / node-api-compatibility-tests 51s 🟩
permission-check 41s 🟩🟩🟩🟩🟩 (+9 more)
permission-check 41s 🟩🟩🟩🟩 (+12 more)
permission-check 39s 🟩🟩🟩🟩 (+12 more)
determine-docker-build-metadata 31s 🟩🟩🟩🟩🟩 (+8 more)
execution-performance / file_change_determinator 11s 🟩
determine-test-metadata 8s 🟩
execution-performance / sequential-execution-performance 7s 🟩
execution-performance / parallel-execution-performance 7s 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job Duration vs 7d avg Delta
forge-framework-upgrade-test / forge 44m 28m +58%

settingsfeedbackdocs ⋅ learn more about trunk.io

@msmouse
Copy link
Contributor Author

msmouse commented Dec 12, 2023

Copy link

codecov bot commented Dec 12, 2023

Codecov Report

Attention: 31 lines in your changes are missing coverage. Please review.

Comparison is base (ce30868) 68.7% compared to head (3c0ca5c) 68.6%.

Files Patch % Lines
...aptos-gas-schedule/src/gas_schedule/transaction.rs 0.0% 22 Missing ⚠️
...ptos-move/aptos-vm-types/src/storage/io_pricing.rs 81.0% 7 Missing ⚠️
aptos-move/aptos-vm-types/src/storage/mod.rs 0.0% 1 Missing ⚠️
aptos-move/framework/table-natives/src/lib.rs 83.3% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             main   #11300     +/-   ##
=========================================
- Coverage    68.7%    68.6%   -0.1%     
=========================================
  Files         764      764             
  Lines      176947   176995     +48     
=========================================
- Hits       121722   121583    -139     
- Misses      55225    55412    +187     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@msmouse msmouse force-pushed the 1211-alden-io-gas branch 2 times, most recently from b05127f to 9cf068a Compare December 12, 2023 17:25
@msmouse msmouse marked this pull request as ready for review December 12, 2023 19:01
@msmouse msmouse requested review from vgao1996, igor-aptos and a team December 12, 2023 19:01
Base automatically changed from 1206_refactor_pricing to main December 12, 2023 19:34
Comment on lines 201 to 205
if self.feature_version >= 12 {
size.checked_sub(self.legacy_free_write_bytes_quota)
.unwrap_or(NumBytes::zero())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it be the opposite?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:trollface: :trollface: :trollface:
👍 👍 👍

impl StoragePricingV3 {
impl IoPricingV3 {
fn read_size(&self, loaded: NumBytes) -> NumBytes {
if self.feature_version >= 12 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you have multiple places with if self.feature_version >= 12, what's the logic when you introduce new version (i.e. IoPricingV4) vs having these conditions inside?

Copy link
Contributor Author

@msmouse msmouse Dec 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if I added v4 it complains that v3 and v4 returns different opaque types here, although they are both STORAGE_IO_PER_STATE_SLOT_READ * NumArgs + STORAGE_IO_PER_STATE_BYTE_READ * NumBytes only differing in the specific numbers inside of NumArgs and NumBytes:

V3(v3) => Either::Right(v3.calculate_read_gas(bytes_loaded)),

I got lazy and just updated v3..
cc @vgao1996

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I can do

   V3(v3) => Either::Right(Either::Left(v3.calculate_read_gas(bytes_loaded))),
   V4(v4) => Either::Right(Either::Right(v4.calculate_read_gas(bytes_loaded)))

?

Do you prefer that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since some params are not part of "V4" , I prefer having separate V3 and V4 in this case

@@ -287,7 +287,17 @@ fn charge_load_cost(

match loaded {
Some(Some(num_bytes)) => {
context.charge(COMMON_LOAD_BASE_NEW + COMMON_LOAD_PER_BYTE * num_bytes)
if context.gas_feature_version() >= 12 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stupid question, why does this not go through IoPricing utilities, like everything else?

Copy link
Contributor Author

@msmouse msmouse Dec 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just keeping the structure..
Indeed it'll be nice if we can remove this shit as well:

match gas_feature_version {
0..=1 => (),
2..=6 => {
if let IoPricing::V2(pricing) = &storage_gas_params.io_pricing {
g.common_load_base_legacy = pricing.per_item_read * NumArgs::new(1);
g.common_load_base_new = 0.into();
g.common_load_per_byte = pricing.per_byte_read;
g.common_load_failure = 0.into();
}
}
7..=9 => {
if let IoPricing::V2(pricing) = &storage_gas_params.io_pricing {
g.common_load_base_legacy = 0.into();
g.common_load_base_new = pricing.per_item_read * NumArgs::new(1);
g.common_load_per_byte = pricing.per_byte_read;
g.common_load_failure = 0.into();
}
}
10.. => {
g.common_load_base_legacy = 0.into();
g.common_load_base_new = gas_params.vm.txn.storage_io_per_state_slot_read * NumArgs::new(1);
g.common_load_per_byte = gas_params.vm.txn.storage_io_per_state_byte_read;
g.common_load_failure = 0.into();
}
};
Ok(storage_gas_params)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you at least add a TODO?

seems like it is very easy to miss this place when modifying gas schedule.

@msmouse msmouse force-pushed the 1211-alden-io-gas branch 2 times, most recently from ae4b209 to 24bdab5 Compare December 12, 2023 22:41
@msmouse msmouse changed the base branch from main to 1206_consolidate_to_pricing December 13, 2023 00:02
Copy link
Contributor

@igor-aptos igor-aptos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks , looks much cleaner!

@@ -287,7 +287,17 @@ fn charge_load_cost(

match loaded {
Some(Some(num_bytes)) => {
context.charge(COMMON_LOAD_BASE_NEW + COMMON_LOAD_PER_BYTE * num_bytes)
if context.gas_feature_version() >= 12 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you at least add a TODO?

seems like it is very easy to miss this place when modifying gas schedule.

Base automatically changed from 1206_consolidate_to_pricing to main December 13, 2023 00:35
1. read bytes charged in 4k steps
2. remove per write op free quota, while make per byte write cheaper

tune write gas numbers

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite compat success on aptos-node-v1.7.3 ==> 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c

Compatibility test results for aptos-node-v1.7.3 ==> 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c (PR)
1. Check liveness of validators at old version: aptos-node-v1.7.3
compatibility::simple-validator-upgrade::liveness-check : committed: 4988 txn/s, latency: 6484 ms, (p50: 6600 ms, p90: 9700 ms, p99: 11200 ms), latency samples: 184580
2. Upgrading first Validator to new version: 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1840 txn/s, latency: 15628 ms, (p50: 19700 ms, p90: 22000 ms, p99: 22500 ms), latency samples: 92040
3. Upgrading rest of first batch to new version: 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1517 txn/s, latency: 15974 ms, (p50: 18800 ms, p90: 22100 ms, p99: 22400 ms), latency samples: 92540
4. upgrading second batch to new version: 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2732 txn/s, latency: 8885 ms, (p50: 9700 ms, p90: 12500 ms, p99: 13300 ms), latency samples: 142100
5. check swarm health
Compatibility test for aptos-node-v1.7.3 ==> 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c passed
Test Ok

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c

two traffics test: inner traffic : committed: 8196 txn/s, latency: 4785 ms, (p50: 4500 ms, p90: 5700 ms, p99: 11700 ms), latency samples: 3532640
two traffics test : committed: 100 txn/s, latency: 2447 ms, (p50: 2100 ms, p90: 2700 ms, p99: 15600 ms), latency samples: 1840
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.270, avg: 0.208", "QsPosToProposal: max: 0.147, avg: 0.135", "ConsensusProposalToOrdered: max: 0.569, avg: 0.545", "ConsensusOrderedToCommit: max: 0.468, avg: 0.443", "ConsensusProposalToCommit: max: 1.021, avg: 0.989"]
Max round gap was 1 [limit 4] at version 1616324. Max no progress secs was 5.188359 [limit 10] at version 1616324.
Test Ok

@msmouse msmouse merged commit 0e91537 into main Dec 13, 2023
99 of 126 checks passed
@msmouse msmouse deleted the 1211-alden-io-gas branch December 13, 2023 07:56
Copy link
Contributor

❌ Forge suite framework_upgrade failure on aptos-node-v1.7.3 ==> 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c

Compatibility test results for aptos-node-v1.7.3 ==> 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c (PR)
Upgrade the nodes to version: 3c0ca5cd9bcec2ab24faa4e5503e34f8ebcb931c
Test Failed: API error: Unknown error error sending request for url (http://aptos-node-3-validator.forge-framework-upgrade-pr-11300.svc:8080/v1/estimate_gas_price): error trying to connect: dns error: failed to lookup address information: Name or service not known

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: __libc_start_main
  13: <unknown>
Trailing Log Lines:
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: __libc_start_main
  13: <unknown>


Swarm logs can be found here: See fgi output for more information.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ApiError: namespaces "forge-framework-upgrade-pr-11300" not found: NotFound (ErrorResponse { status: "Failure", message: "namespaces \"forge-framework-upgrade-pr-11300\" not found", reason: "NotFound", code: 404 })

Caused by:
    namespaces "forge-framework-upgrade-pr-11300" not found: NotFound

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: __libc_start_main
  16: <unknown>', testsuite/forge/src/backend/k8s/swarm.rs:676:18
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants