Automate version comparison #260
Conversation
There was a problem hiding this comment.
Hi @ikabadzhov , it looks like you added too many files to this commit, including some under gbench? Also can we directly use the compare.py that comes with gbench rather than copying it into rootbench?
|
|
||
| Example usage: | ||
| ``` | ||
| sh rootbench_compare.sh -b latest-stable -c 9e7933c9bddba3f6381ba2b8c11539c46e8cc2f4 \ |
There was a problem hiding this comment.
| sh rootbench_compare.sh -b latest-stable -c 9e7933c9bddba3f6381ba2b8c11539c46e8cc2f4 \ | |
| bash rootbench_compare.sh -b latest-stable -c 9e7933c9bddba3f6381ba2b8c11539c46e8cc2f4 \ |
| 2. "b" - branch name of the baseline (default is `master`) | ||
| 3. "c" - commit hash of the baseline (if none provided, the latest commit is picked) | ||
| 4. "R" - owner of the baseline (default is `root-project`) | ||
| 5. "B" - branch name of the baseline (default is `master`) | ||
| 6. "C" - commit hash of the baseline (if none provided, the latest commit is picked) | ||
| 7. "t" - regex to be passed to rootbench ctest (default empty string, hence all tests) | ||
| 8. "n" - cores to use (default empty string, hence all cores of the machine) |
There was a problem hiding this comment.
is it intentional that the description of b/B, c/C and r/R are the same?
| 2. "b" - branch name of the baseline (default is `master`) | ||
| 3. "c" - commit hash of the baseline (if none provided, the latest commit is picked) |
There was a problem hiding this comment.
shouldn't branch name and commit hash be mutually exclusive?
| @@ -0,0 +1,91 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
| git reset --hard ${commit_baseline} | ||
| fi | ||
| mkdir rbuild && cd rbuild | ||
| cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-O2 -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" .. |
There was a problem hiding this comment.
| cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-O2 -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" .. | |
| cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" .. |
| git reset --hard ${commit_compared} | ||
| fi | ||
| mkdir rbuild && cd rbuild | ||
| cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-O2 -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" .. |
There was a problem hiding this comment.
| cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-O2 -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" .. | |
| cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" .. |
|
|
||
| source baseline_root/rbuild/bin/thisroot.sh | ||
| mkdir baseline_bench && cd baseline_bench | ||
| cmake -Drootbench-datafiles=ON -Drootbench-out-format-json=ON ../../.. |
There was a problem hiding this comment.
this assumes that the script is being executed from the root directory of the rootbench repo. we should probably check that this is the case at the very beginning of the script.
| esac | ||
| done | ||
|
|
||
| git clone https://github.com/${root_owner_baseline}/root --branch ${branch_baseline} baseline_root |
There was a problem hiding this comment.
we should only git clone if the repository is not already present. also consider performing a sparse checkout (but I don't know how that interacts with executions of this script after the first)
| @@ -0,0 +1,429 @@ | |||
| #!/usr/bin/env python | |||
There was a problem hiding this comment.
I thought compare.py comes with standard google benchmark. Is this compare.py different?
|
Thanks for working on this. That's really exciting. Could you add documentation/script which demonstrates how to use these tools to do |
|
Hi Vassil,
This is not something you can do with this script as it is, might be something that can be built on top in a future PR |
This is a small script to compare performance between 2 different patches/commits.
The script will take care of building 2 ROOT versions and running the desired benchmarks.
And finally https://github.com/google/benchmark/blob/main/tools/compare.py compares both.