Home

(Adapted from https://docs.google.com/document/d/1AoPBAbD8DiDVEM6HuOtPKekN3phtcCF4Qk6pxZ0ID-w/edit)

Note: asv does have documentation here but the present document are just some quick instructions on getting set up with asv for poliastro. Please help make this document clearer, as we can then eventually put it in the poliastro docs!

Getting set up

Before you can run the poliastro benchmarks with asv you need to install asv:

pip install asv

and clone the benchmarks repository:

git clone [email protected]:poliastro/poliastro-benchmarks.git
cd poliastro-benchmarks

Writing benchmarks

Benchmarks are written as Python functions which are stored inside the benchmarks directory. This directory should contain one file per poliastro sub-package. Benchmarks should be kept as simple as absolutely possible, in many cases a single line.

Trying out benchmarks

The easiest/fastest way to try out the benchmarks is to make sure you have poliastro installed (either properly installed or just in ‘develop’ mode), then run:

asv dev

This will run the benchmarks against the local poliastro version and will do some basic timing, but the timing will not be very accurate, because this only runs each benchmark once instead of taking an average of many runs. Nevertheless, this is the first step to make sure things are running correctly and can still give order of magnitude timings.

Running asv properly

To run asv on the latest commit in the upstream poliastro master, you can do:

asv run

This will set up a temporary environment in which poliastro will be installed, and the benchmark functions will be run multiple times and averaged to get accurate timings. The results from this will be saved into a results/<machine-name> directory, which will store one file per commit (or more than one file if the tests are set up to run for multiple python or numpy versions, which they are not configured to do by default for poliastro).

Now in practice when using asv locally you will probably want to be comparing the latest master to your latest developer version to see if the changes you are making are improving things or making things worse. So we should switch from using the upstream repository to the repository on your computer. To do this, edit the asv.conf.json file and find the following section:

    // The git URL to the project being tested. Comment the first line
    // and uncomment and edit the second if you are testing local changes.
    "repo": "https://github.com/poliastro/poliastro.git",
    //"repo": "/your/local/repository/”,

Comment out the first "repo" line and uncomment the second, replacing the path with the absolute path to your local clone of poliastro. Then run e.g.

asv run 827f322b^!

replacing 827f322b by the commit you want to test (the ^! Indicates to just run this commit, not all commits up to that point). If you want to test out the latest changes you’ve made, the best way to not get confused is to commit your changes (locally) then run the same command but with the updated commit hash. If you want to run a range of commits, use:

asv run 827f322b..729abcf3

Selecting benchmarks to run

Running all benchmarks can take a while, so you can specify which ones to run with the --bench/-b option (which can even take a regular expression):

asv run --bench write

You can combine this with the syntax above to run the benchmarks for a specific commit, e.g.:

asv run --bench CsvFloat 827f322b^!

Comparing results in your branch to poliastro master

Once you have run the tests for more than one commit, you can compare these using e.g.:

asv compare 827f322b

Speeding things up

Using asv run can be slow, but can be sped up a bit by using asv run --quick (which runs each benchmark only once). Beyond this, you can also just use asv dev but the results aren’t saved and you can’t use asv compare, but you could simply visually compare the output of the two commands. Setting up the benchmarks to run regularly

Asv can be run as a cron job and the results can be pushed to GitHub. Since you won't be able to run all the commits since the start of the project in one go, start off by running the benchmarks for the current commit.

asv run HEAD^! [-c 0]

Then commit the new files in results/ to the repository:

git add results
git commit -m 'Initial result from new machine'
git push origin master

Once this is done, you can then set up a cron job that will run:

asv run NEW [-c 0]

It makes sense to make this a daily cron job (no more frequent)

In addition you could consider including a command in the cron job to run older commits:

timeout 7200 asv run ALL [-c 0] --skip-existing-commits

By default this would run all commits that haven’t already been run, but the timeout command ensures this doesn’t work for too long, and is finished before the next cron job (the benefit to interrupting this regularly is to be able to commit the changes and make sure we also run new commits as they are made to the repository). The timeout value is in seconds and can be adjusted.

Make sure that at the end of each cron job you commit the results to the repository and publish a new version of the website. Also assume that other people may be running the benchmarks, and therefore you will need to pull from the benchmarks repo before pushing.

We have a cron.sh file in the poliastro-benchmarks repo that can do the above, though please double check it before using it!