Benchmark runner script (#5215)

* WIP.

* WIP.

* WIP.

* Keep Python versions aligned via a test.

* Completed benchmark runner.

* Remove Nox benchmark session.

* Note on Conda package cache.

* bm_runner licence header.

* Bump env cache build.

* Bump nox cache build.

* Revert "Bump env cache build."

This reverts commit 5a8946c.

* Remove extraneous pass command.

* bm_runner subparsers required=True.

* bm_runner examples.

* What's New entry.

* Clarify co-location of benchmark env and Conda package cache.

.github/workflows/benchmark.yml

@@ -25,17 +25,16 @@ jobs:
       # Lets us manually bump the cache to rebuild
       ENV_CACHE_BUILD: "0"
       TEST_DATA_CACHE_BUILD: "2"
-      PY_VER: 3.8
 
     steps:
       # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
       - uses: actions/checkout@v3
         with:
           fetch-depth: 0
 
-      - name: Install Nox
+      - name: Install ASV & Nox
         run: |
-          pip install nox
+          pip install asv nox
 
       - name: Cache environment directories
         id: cache-env-dir
@@ -78,7 +77,7 @@ jobs:
           
           if [ "$first_commit" != "" ]
           then
-            nox --session="benchmarks(overnight)" -- $first_commit
+            python benchmarks/bm_runner.py overnight $first_commit
           fi
 
       - name: Create issues for performance shifts

.github/workflows/ci-tests.yml

@@ -111,7 +111,7 @@ jobs:
       - name: "nox cache"
         uses: ./.github/workflows/composite/nox-cache
         with:
-          cache_build: 0
+          cache_build: 1
           env_name: ${{ env.ENV_NAME }}
           lock_file: ${{ env.LOCK_FILE }}
 

benchmarks/README.md

@@ -15,9 +15,20 @@ shifts in performance being flagged in a new GitHub issue.
 installed, as well as Nox (see
 [Benchmark environments](#benchmark-environments)).
 
-[Iris' noxfile](../noxfile.py) includes a `benchmarks` session that provides
-conveniences for setting up before benchmarking, and can also replicate the
-automated overnight run locally. See the session docstring for detail.
+The benchmark runner ([bm_runner.py](./bm_runner.py)) provides conveniences for
+common benchmark setup and run tasks, including replicating the automated 
+overnight run locally. See `python bm_runner.py --help` for detail.
+
+A significant portion of benchmark run time is environment management. Run-time
+can be reduced by placing the benchmark environment on the same file system as
+your
+[Conda package cache](https://conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html#specify-pkg-directories),
+if it is not already. You can achieve this by either:
+
+- Temporarily reconfiguring `delegated_env_commands` and `delegated_env_parent` 
+  in [asv.conf.json](asv.conf.json) to reference a location on the same file
+  system as the Conda package cache.
+- Moving your Iris repo to the same file system as the Conda package cache.
 
 ### Environment variables
 
@@ -26,8 +37,8 @@ automated overnight run locally. See the session docstring for detail.
 benchmark scripts.
 * `DATA_GEN_PYTHON` - required - path to a Python executable that can be
 used to generate benchmark test objects/files; see
-[Data generation](#data-generation). The Nox session sets this automatically,
-but will defer to any value already set in the shell.
+[Data generation](#data-generation). The benchmark runner sets this 
+automatically, but will defer to any value already set in the shell.
 * `BENCHMARK_DATA` - optional - path to a directory for benchmark synthetic
 test data, which the benchmark scripts will create if it doesn't already
 exist. Defaults to `<root>/benchmarks/.data/` if not set. Note that some of
@@ -36,7 +47,7 @@ plan accordingly.
 * `ON_DEMAND_BENCHMARKS` - optional - when set (to any value): benchmarks
 decorated with `@on_demand_benchmark` are included in the ASV run. Usually
 coupled with the ASV `--bench` argument to only run the benchmark(s) of
-interest. Is set during the Nox `cperf` and `sperf` sessions.
+interest. Is set during the benchmark runner `cperf` and `sperf` sub-commands.
 
 ## Writing benchmarks
 

benchmarks/asv.conf.json

@@ -19,8 +19,7 @@
     //  * No build-time environment variables.
     //  * Is run in the same environment as the ASV install itself.
     "delegated_env_commands": [
-        "sed -i 's/_PY_VERSIONS_ALL/_PY_VERSION_LATEST/g' noxfile.py",
-        "nox --envdir={conf_dir}/.asv/env/nox01 --session=tests --install-only --no-error-on-external-run --verbose"
+        "PY_VER=3.10 nox --envdir={conf_dir}/.asv/env/nox01 --session=tests --install-only --no-error-on-external-run --verbose"
     ],
     // The parent directory of the above environment.
     //  The most recently modified environment in the directory will be used.