Skip to content

Commit

Permalink
Restore run-benchmarks action
Browse files Browse the repository at this point in the history
  • Loading branch information
@fmiguelgarcia
fmiguelgarcia committed Oct 31, 2023
1 parent c88616a commit 9630426
Showing 1 changed file with 20 additions and 57 deletions.
77 changes: 20 additions & 57 deletions .github/workflows/run-benchmarks.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,6 @@ on:
description: Benchmark only our pallet
required: false
default: 0
env:
ACTIONS_RUNNER_DEBUG: true
ACTIONS_STEP_DEBUG: true

jobs:
benchmark:
Expand All @@ -24,7 +21,8 @@ jobs:
- name: Install deps
run: |
sudo apt-get update
sudo apt-get install -y build-essential valgrind git clang curl libssl-dev protobuf-compiler unzip
sudo apt-get install -y build-essential
sudo apt-get install -y git clang curl libssl-dev protobuf-compiler unzip
- name: Install Protoc
uses: arduino/setup-protoc@v1
Expand All @@ -41,61 +39,26 @@ jobs:
- name: Set PATH for cargo
run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH

# - name: Build node and run benchmarks
# run: |
# if [ "${{ github.event.inputs.extra }}" != "0" ]; then
# EXTRA="EXTRA=${{ github.event.inputs.extra }}"
# else
# EXTRA=""
# fi

# if [ "${{ github.event.inputs.ourpallets }}" != "0" ]; then
# OUR_PALLETS="OUR_PALLETS=${{ github.event.inputs.ourpallets }}"
# else
# OUR_PALLETS=""
# fi

# echo "Command to be executed: $EXTRA $OUR_PALLETS ./scripts/run_benchmarks.sh"

# $EXTRA $OUR_PALLETS ./scripts/run_benchmarks.sh
- name: Build node and run benchmarks
run: |
if [ "${{ github.event.inputs.extra }}" != "0" ]; then
EXTRA="EXTRA=${{ github.event.inputs.extra }}"
else
EXTRA=""
fi
# - name: Upload output as artifact
# uses: actions/upload-artifact@v2
# with:
# name: weights-result
# path: ./output/
if [ "${{ github.event.inputs.ourpallets }}" != "0" ]; then
OUR_PALLETS="OUR_PALLETS=${{ github.event.inputs.ourpallets }}"
else
OUR_PALLETS=""
fi
# Run Header Generation benchmarks
- name: Download previous Header Generation Benchmark
uses: actions/cache@v3
with:
path: |
./cache
./target/release/
./runtime/target/iai
key: ${{ runner.os }}-header-gen-benchmark
echo "Command to be executed: $EXTRA $OUR_PALLETS ./scripts/run_benchmarks.sh"
- name: Run Header Generation Benchmark
run: cargo bench -p da-runtime | tee ./header_gen_bench.txt
$EXTRA $OUR_PALLETS ./scripts/run_benchmarks.sh
- name: Header Generation Regression Checks
uses: fmiguelgarcia/github-action-benchmark@v1
- name: Upload output as artifact
uses: actions/upload-artifact@v2
with:
# What benchmark tool the output.txt came from
tool: 'rustIai'
# Where the output from the benchmark tool is stored
# output-file-path: /home/ubuntu/actions-runner/_work/avail/avail/header_gen_bench.txt
output-file-path: ./header_gen_bench.txt
# Where the previous data file is stored
external-data-json-path: ./cache/benchmark-data.json
# Workflow will fail when an alert happens at 15% degradation
fail-on-alert: true
alert-threshold: '15%'
# Upload the updated cache file for the next job by actions/cache
github-token: ${{ secrets.GITHUB_TOKEN }}
# Enable alert commit comment
comment-on-alert: true
# Mention @rhysd in the commit comment
alert-comment-cc-users: '@fmiguelgarcia,@prabal-banerjee,@jakubcech,@vthunder,@kroos47,@Leouarz'
comment-always: true
summary-always: true
name: weights-result
path: ./output/

3 comments on commit 9630426

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark

Benchmark suite Current: 9630426 Previous: 632d8f9 Ratio
commitment_builder/32 4339018471 ns/iter (± 98870394) 4239305043 ns/iter (± 103025063) 1.02
commitment_builder/64 4389647317 ns/iter (± 199693608) 4146927392 ns/iter (± 96859072) 1.06
commitment_builder/128 4783517620 ns/iter (± 183411326) 4527921057 ns/iter (± 59996064) 1.06
commitment_builder/256 2399982167 ns/iter (± 101137907) 2205138657 ns/iter (± 36843535) 1.09

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark

Benchmark suite Current: 9630426 Previous: 632d8f9 Ratio
commitment_builder_32 Instructions 43503524573 ops (± 0) 42920265841 ops (± 0) 1.01
commitment_builder_32 L1 Accesses 53515411837 ops (± 0) 52858711440 ops (± 0) 1.01
commitment_builder_32 L2 Accesses 733942 ops (± 0) 430583 ops (± 0) 1.70
commitment_builder_32 RAM Accesses 91408 ops (± 0) 101017 ops (± 0) 0.90
commitment_builder_32 Estimated Cycles 53522280827 ops (± 0) 52864399950 ops (± 0) 1.01
commitment_builder_64 Instructions 41972904675 ops (± 0) 41390334795 ops (± 0) 1.01
commitment_builder_64 L1 Accesses 51407926800 ops (± 0) 50751723761 ops (± 0) 1.01
commitment_builder_64 L2 Accesses 894547 ops (± 0) 555435 ops (± 0) 1.61
commitment_builder_64 RAM Accesses 190833 ops (± 0) 199375 ops (± 0) 0.96
commitment_builder_64 Estimated Cycles 51419078690 ops (± 0) 50761479061 ops (± 0) 1.01
commitment_builder_128 Instructions 54204654417 ops (± 0) 53619537848 ops (± 0) 1.01
commitment_builder_128 L1 Accesses 68302593854 ops (± 0) 67644600246 ops (± 0) 1.01
commitment_builder_128 L2 Accesses 1316411 ops (± 0) 1015589 ops (± 0) 1.30
commitment_builder_128 RAM Accesses 245023 ops (± 0) 269618 ops (± 0) 0.91
commitment_builder_128 Estimated Cycles 68317751714 ops (± 0) 67659114821 ops (± 0) 1.01
commitment_builder_256 Instructions 31758788295 ops (± 0) 31178642102 ops (± 0) 1.02
commitment_builder_256 L1 Accesses 40833619944 ops (± 0) 40169352892 ops (± 0) 1.02
commitment_builder_256 L2 Accesses 6801151 ops (± 0) 6485426 ops (± 0) 1.05
commitment_builder_256 RAM Accesses 236286 ops (± 0) 268224 ops (± 0) 0.88
commitment_builder_256 Estimated Cycles 40875895709 ops (± 0) 40211167862 ops (± 0) 1.02

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.15.

Benchmark suite Current: 9630426 Previous: 632d8f9 Ratio
commitment_builder_32 L2 Accesses 733942 ops (± 0) 430583 ops (± 0) 1.70
commitment_builder_64 L2 Accesses 894547 ops (± 0) 555435 ops (± 0) 1.61
commitment_builder_128 L2 Accesses 1316411 ops (± 0) 1015589 ops (± 0) 1.30

This comment was automatically generated by workflow using github-action-benchmark.

CC: @prabal-banerjee @jakubcech @vthunder @kroos47 @Leouarz @markopoloparadox

Please sign in to comment.