Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix benchmark reporting with subtitles of result key #590

Merged
merged 10 commits into from
Jun 17, 2024
Merged

Fix benchmark reporting with subtitles of result key #590

merged 10 commits into from
Jun 17, 2024

Conversation

afsalthaj
Copy link
Contributor

No description provided.

@afsalthaj
Copy link
Contributor Author

/run-benchmark-refresh

afsalthaj
afsalthaj commented Jun 15, 2024
View reviewed changes
integration-tests/src/benchmarks/report/benchmark_report.rs
run_config_to_avg_time.insert(run_config.clone(), avg);
} else {
run_config_to_avg_time.insert(run_config.clone(), Duration::ZERO);
pub fn calculate_mean_avg_time(json: &BenchmarkResult) -> HashMap<ReportKey, Duration> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change this name

@github-actions GitHub Actions
Copy link

Benchmark Report

benchmark_cold_start_large.json

Result Key Cluster Size Size Length Avg Time
worker-1 3 10 100 202.036752ms
worker-4 3 10 100 202.859404ms
worker-7 3 10 100 203.642377ms
worker-0 3 10 100 197.547797ms
worker-2 3 10 100 198.287018ms
worker-8 3 10 100 200.346415ms
invocation 3 10 100 200.351575ms
worker-3 3 10 100 199.057685ms
worker-9 3 10 100 198.708075ms
worker-6 3 10 100 201.459484ms
worker-5 3 10 100 199.570743ms

benchmark_cold_start_large_no_compilation.json

Result Key Cluster Size Size Length Avg Time
worker-2 3 10 100 114.167724279s
worker-0 3 10 100 114.165812503s
worker-7 3 10 100 114.16805017s
worker-9 3 10 100 114.16843051s
invocation 3 10 100 114.16877052s
worker-6 3 10 100 114.169325466s
worker-3 3 10 100 114.171364685s
worker-1 3 10 100 114.168034386s
worker-5 3 10 100 114.170081094s
worker-4 3 10 100 114.17014437s
worker-8 3 10 100 114.168737738s

benchmark_cold_start_medium.json

Result Key Cluster Size Size Length Avg Time
worker-2 3 10 100 137.640741ms
worker-0 3 10 100 132.08493ms
invocation 3 10 100 136.623096ms
worker-8 3 10 100 139.025908ms
worker-3 3 10 100 139.257167ms
worker-7 3 10 100 138.152439ms
worker-6 3 10 100 135.223761ms
worker-9 3 10 100 132.142185ms
worker-4 3 10 100 139.313087ms
worker-1 3 10 100 137.671712ms
worker-5 3 10 100 135.719028ms

benchmark_cold_start_medium_no_compilation.json

Result Key Cluster Size Size Length Avg Time
worker-8 3 10 100 169.424334019s
worker-6 3 10 100 169.421452384s
worker-4 3 10 100 169.422705084s
worker-0 3 10 100 169.422376816s
worker-9 3 10 100 169.423016701s
worker-3 3 10 100 169.421131939s
worker-1 3 10 100 169.421927157s
worker-2 3 10 100 169.42132723s
invocation 3 10 100 169.42190693s
worker-5 3 10 100 169.419402253s
worker-7 3 10 100 169.421395716s

benchmark_cold_start_small.json

Result Key Cluster Size Size Length Avg Time
worker-6 3 10 100 76.629431ms
worker-1 3 10 100 79.985282ms
worker-0 3 10 100 75.92644ms
worker-5 3 10 100 79.496835ms
worker-9 3 10 100 75.516707ms
invocation 3 10 100 78.427965ms
worker-3 3 10 100 83.188272ms
worker-4 3 10 100 78.495766ms
worker-8 3 10 100 79.53819ms
worker-2 3 10 100 77.782508ms
worker-7 3 10 100 77.720217ms

benchmark_cold_start_small_no_compilation.json

Result Key Cluster Size Size Length Avg Time
worker-2 3 10 100 360.188761ms
worker-8 3 10 100 359.539792ms
worker-5 3 10 100 359.158635ms
invocation 3 10 100 360.066704ms
worker-7 3 10 100 361.034471ms
worker-9 3 10 100 362.112327ms
worker-1 3 10 100 359.294041ms
worker-3 3 10 100 359.889716ms
worker-6 3 10 100 359.514241ms
worker-4 3 10 100 360.564603ms
worker-0 3 10 100 359.370455ms

benchmark_durability_overhead.json

Result Key Cluster Size Size Length Avg Time
durable-worker-0 3 10 100 58.199614ms
not-durable-worker-2 3 10 100 58.427791ms
durable-worker-8 3 10 100 58.454191ms
durable-worker-4 3 10 100 57.662938ms
not-durable-worker-8 3 10 100 58.465303ms
not-durable-worker-4 3 10 100 58.163547ms
not-durable-worker-3 3 10 100 57.401717ms
not-durable-worker-7 3 10 100 58.135212ms
durable-worker-2 3 10 100 57.981719ms
not-durable-worker-0 3 10 100 58.183091ms
durable-worker-5 3 10 100 58.723299ms
not-durable-invocation 3 10 100 58.241441ms
durable-worker-9 3 10 100 58.223285ms
durable-worker-7 3 10 100 57.510528ms
durable-worker-6 3 10 100 57.95525ms
durable-worker-1 3 10 100 58.11899ms
durable-invocation 3 10 100 58.082365ms
not-durable-worker-5 3 10 100 58.635219ms
durable-worker-3 3 10 100 57.993836ms
not-durable-worker-6 3 10 100 58.749842ms
not-durable-worker-9 3 10 100 58.238544ms
not-durable-worker-1 3 10 100 58.014146ms

benchmark_latency_large.json

Result Key Cluster Size Size Length Avg Time
worker-9 3 10 100 65.538645ms
worker-3 3 10 100 65.03865ms
worker-2 3 10 100 64.977827ms
worker-0 3 10 100 65.153383ms
worker-4 3 10 100 65.379063ms
worker-6 3 10 100 65.132611ms
worker-7 3 10 100 65.288174ms
worker-5 3 10 100 65.316962ms
worker-1 3 10 100 64.554211ms
invocation 3 10 100 65.116828ms
worker-8 3 10 100 64.788755ms

benchmark_latency_medium.json

Result Key Cluster Size Size Length Avg Time
worker-0 3 10 100 57.712659ms
worker-7 3 10 100 57.989348ms
worker-3 3 10 100 58.124963ms
invocation 3 10 100 57.916199ms
worker-1 3 10 100 57.879711ms
worker-5 3 10 100 57.885342ms
worker-8 3 10 100 57.446443ms
worker-9 3 10 100 58.166427ms
worker-4 3 10 100 58.227153ms
worker-2 3 10 100 57.787475ms
worker-6 3 10 100 57.942468ms

benchmark_latency_small.json

Result Key Cluster Size Size Length Avg Time
worker-9 3 10 100 61.929731ms
worker-5 3 10 100 61.843442ms
worker-2 3 10 100 61.917628ms
worker-0 3 10 100 61.950627ms
invocation 3 10 100 61.819236ms
worker-8 3 10 100 61.872087ms
worker-4 3 10 100 61.902269ms
worker-3 3 10 100 61.517083ms
worker-7 3 10 100 61.843853ms
worker-6 3 10 100 61.523313ms
worker-1 3 10 100 61.892331ms

benchmark_suspend_worker.json

Result Key Cluster Size Size Length Avg Time
worker-2 3 10 100 10.028817606s
worker-0 3 10 100 10.029823978s
worker-6 3 10 100 10.025546282s
invocation 3 10 100 10.029814976s
worker-8 3 10 100 10.029084621s
worker-7 3 10 100 10.028182289s
worker-3 3 10 100 10.032208953s
worker-1 3 10 100 10.032802178s
worker-4 3 10 100 10.032419076s
worker-5 3 10 100 10.027036209s
worker-9 3 10 100 10.032228563s

RunID: 9530155111

@afsalthaj
Copy link
Contributor Author

/run-benchmark

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 16, 2024 •
edited by afsalthaj
Loading

Benchmark Comparison Report

benchmark_cold_start_large.json

ResultKey Cluster Size Size Length Previous Avg Time Current Avg Time
worker-5 3 10 100 237.491242ms 183.297241ms
worker-0 3 10 100 234.030021ms 191.123109ms
worker-7 3 10 100 228.847621ms 194.449385ms
worker-6 3 10 100 233.101302ms 186.764602ms
worker-9 3 10 100 230.217783ms 188.088617ms
worker-1 3 10 100 221.491274ms 174.941248ms
worker-8 3 10 100 223.023152ms 190.270198ms
worker-3 3 10 100 220.393216ms 188.059451ms
worker-2 3 10 100 222.230051ms 192.62944ms
invocation 3 10 100 227.757226ms 186.815966ms
worker-4 3 10 100 226.746597ms 178.536375ms

benchmark_cold_start_large_no_compilation.json

ResultKey Cluster Size Size Length Previous Avg Time Current Avg Time
worker-4 3 10 100 114.598561586s 115.422807306s
worker-3 3 10 100 114.596261814s 115.422627038s
worker-7 3 10 100 114.59798122s 115.424090079s
worker-2 3 10 100 114.601882776s 115.422753134s
worker-5 3 10 100 114.593268031s 115.420340065s
worker-8 3 10 100 114.599797421s 115.425995818s
worker-0 3 10 100 114.599811738s 115.423681115s
worker-6 3 10 100 114.59910207s 115.422566721s
worker-1 3 10 100 114.597722083s 115.42036951s
worker-9 3 10 100 114.595676644s 115.42392282s
invocation 3 10 100 114.598006538s 115.42291536s

....

RunID: 9532511663

@afsalthaj
Copy link
Contributor Author

/run-benchmark

@afsalthaj afsalthaj merged commit 23c7035 into main Jun 17, 2024
14 checks passed
@afsalthaj afsalthaj deleted the fix_benchmark branch June 17, 2024 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@justcoon justcoon justcoon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@afsalthaj @justcoon @actions-user