diff --git a/build_tools/pkgci/README.md b/build_tools/pkgci/README.md index 8121ba2a67a7..898da91d1826 100644 --- a/build_tools/pkgci/README.md +++ b/build_tools/pkgci/README.md @@ -1,12 +1,28 @@ # PkgCI Scripts -This directory contains scripts and configuration for the "new" CI, which +This directory contains scripts and configuration for "PkgCI", which is based on building packages and then flowing those to followon jobs. -The traditional CI attempted to do all steps as various kinds of source +The prior/traditional CI attempted to do all steps as various kinds of source builds at head vs a split package/test style of workflow. It can mostly be found in the `cmake` directory but is also scattered around. This directory generally corresponds to "pkgci_" prefixed workflows. Over time, as this CI flow takes over more of the CI pipeline, the traditional CI will be reduced to outlier jobs and policy checks. + +### Development notes + +Testing venv setup using packages: + +```bash +python3.11 ./setup_venv.py /tmp/.venv --fetch-git-ref=5b0740c97a33ed + +# Activate the venvs and test it +source /tmp/.venv/bin/activate +iree-compile --version +# IREE (https://iree.dev): +# IREE compiler version 3.1.0.dev+5b0740c97a33edce29e753b14b9ff04789afcc53 @ 5b0740c97a33edce29e753b14b9ff04789afcc53 +# LLVM version 20.0.0git +# Optimized build +``` diff --git a/build_tools/pkgci/bisect/README.md b/build_tools/pkgci/bisect/README.md new file mode 100644 index 000000000000..ba7e9153b2b1 --- /dev/null +++ b/build_tools/pkgci/bisect/README.md @@ -0,0 +1,329 @@ +# Package bisect scripting + +This scripting connects the `git bisect` tool +(https://git-scm.com/docs/git-bisect) with IREE's package builds, allowing +developers to run tests through commit history efficiently. For example, this +can be used to spot at which commit an `iree-compile` command started failing. + +At each stage of the bisect process, this bisect tool will download the IREE +packages (i.e. `iree-base-compiler` and `iree-base-runtime`) and prepend their +installed location to the `PATH` environment variable. If you want to run a +bisect that _does not_ need to run `iree-compile`, just use `git bisect` +directly. However, if you _do_ need to run `iree-compile`, this can save +substantial time by avoiding the need to build it from source at each test +commit. + +## Prerequisites + +### System requirements + +Requirement | Details +----------- | ------- +Linux | (at least until IREE builds packages for other systems at each commit) +`git` | https://git-scm.com/ +`gh` CLI | https://cli.github.com/ +iree-org/iree repository read access | Needed to [download workflow artifacts](https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/downloading-workflow-artifacts). See also [obtaining commit access](https://iree.dev/developers/general/contributing/#obtaining-commit-access). +`python3.11` with `venv` support | (Version must match what PkgCI builds) `sudo apt install python3.11 python3.11-dev python3.11-venv` + +### Data requirements + +* A command to run, such as a `.mlir` file and a `iree-compile` command +* A known-working commit +* A known-broken commit + +The commit range between known-working and known-broken should be as small as +possible to limit the number of test steps. The bisect algorithm is `O(log N)` +where `N` is the number of commits in the range, but wider windows have larger +risk of something in the test environment breaking (e.g. breaking API changes, +serialized `.mlir` files not being stable, etc.). + +## Usage + +### Example + +Let's try to find the culprit commit for issue +https://github.com/iree-org/iree/issues/18879. Thanks to the detailed issue +description, we have all the data we need to run a bisect already. + +To run the bisect tool: + +1. Setup the test case by saving the input `.mlir` file and constructing a test + `.sh` script: + + ```mlir + // /tmp/issue_18879.mlir + + // This is the input program to the test script. + module { + func.func @main_graph( + // ... + ``` + + ```bash + # /tmp/issue_18879.sh + + # This is the script that will be tested at each commit. + # This script should succeed (return 0) at and prior to the `--good-ref` + # commit and should fail (return non-0) at the `--bad-ref` commit. + + # Try to keep these test scripts minimal. If the failure is in an earlier + # phase of the compiler (e.g. 'Flow' or 'Stream'), consider using + # a flag like `--compile-to=hal` to exit early on successful run instead + # of spending all the time to serialize an output `.vmfb` file. + # https://iree.dev/developers/general/developer-tips/#compiling-phase-by-phase + + iree-compile --iree-hal-target-backends=llvm-cpu -o /dev/null /tmp/issue_18879.mlir + ``` + + ```bash + # Set the script as executable. + chmod +x /tmp/issue_18879.sh + ``` + +2. Run the bisect tool, under Python 3.11: + + ```bash + # Ensure 'python' is Python 3.11, e.g. using venv + # (https://docs.python.org/3/library/venv.html): + python3.11 -m venv .venv && source .venv/bin/activate + # OR using pyenv (https://github.com/pyenv/pyenv): + # pyenv shell 3.11 + python --version + # Python 3.11.10 + + ./bisect_packages.py \ + --good-ref=f9fa934c649749b30fc4be05d9cef78eb043f0e9 \ + --bad-ref=05bbcf1385146d075829cd940a52bf06961614d0 \ + --test-script=/tmp/issue_18879.sh + + # 206b60ca59c9dbbca5769694df4714c38cecaced is the first bad commit + ``` + + As expected, the bisect agrees with the culprit mentioned on the issue: + https://github.com/iree-org/iree/issues/18879#issuecomment-2435531655. + + Note that any git ref can be used, so we can use tags too: + + ```bash + ./bisect_packages.py \ + --good-ref=candidate-20241016.1048 \ + --bad-ref=candidate-20241017.1049 \ + --test-script=/tmp/issue_18879.sh + ``` + +## How the tool works + +1. The [`bisect_packages.py`](./bisect_packages.py) script is the main entry + point which sets things up and runs `git bisect` commands. +2. The bisect operation starts with + `git bisect start --no-checkout --first-parent` (flags to avoid modifying + the working tree and traversing merge commits) and then calls to + specify the commit range with `git bisect good` and `git bisect bad`. +3. The script injects wrapper code around the provided `--test-script`. First, + the wrapper script calls + [`install_packages_for_commit.py`](./install_packages_for_commit.py) to + download IREE packages built at the test commit (marked by `BISECT_HEAD`) and + install those packages into an isolated virtual environment. The wrapper + script then puts that environment at the front of `PATH`, runs the original + script, and finally forwards the original script's exit code to `git bisect`. +4. The script kicks off a `git bisect run` using the generated wrapper script, + which then proceeds to test commits between `--good-ref` and `--bad-ref`, + looking for when the test script switched from succeeding to failing. +5. After the script, the logs can be analyzed and `git bisect log` can be run + from the repository root. + +### Working directory cache + +Downloaded files and virtual environments are cached at `~/.iree/bisect` +(this path can be changed using the `--work-dir` option). Each commit tested +gets its own subfolder that contains the downloaded release artifacts and a +Python venv with those packages installed in it: + +```bash +$ tree -a ~/.iree/bisect -L 2 +/home/nod/.iree/bisect +├── 099ffd556bc5d35efcca32af51cccc061a273a91 +│   ├── iree_base_compiler-3.1.0.dev0+099ffd556bc5d35efcca32af51cccc061a273a91-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl +│   ├── iree_base_runtime-3.1.0.dev0+099ffd556bc5d35efcca32af51cccc061a273a91-cp311-cp311-manylinux_2_28_x86_64.whl +│   └── .venv +├── 15006418ceb03023e8887cba87e93b499f669ad7 +│   ├── iree_compiler-0.dev1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl +│   ├── iree_runtime-0.dev1-cp311-cp311-manylinux_2_28_x86_64.whl +│   └── .venv +├── 206b60ca59c9dbbca5769694df4714c38cecaced +│   ├── iree_compiler-0.dev1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl +│   ├── iree_runtime-0.dev1-cp311-cp311-manylinux_2_28_x86_64.whl +│   └── .venv +├── 23c32c633c01e0237cf5f3815b6647cf01827832 +│   ├── iree_base_compiler-3.1.0.dev0+23c32c633c01e0237cf5f3815b6647cf01827832-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl +│   ├── iree_base_runtime-3.1.0.dev0+23c32c633c01e0237cf5f3815b6647cf01827832-cp311-cp311-manylinux_2_28_x86_64.whl +│   └── .venv +``` + +### Wrapper script + +Here is an example of a script that wraps the original `--test-script`. This is +what gets passed to `git bisect run`: + +```bash +#!/bin/bash + +######################################### +###### BISECT RELEASE SCRIPT SETUP ###### +######################################### + +set -xeuo pipefail + +REF_HASH=$(git rev-parse BISECT_HEAD) +"/home/nod/.pyenv/shims/python3.11" /home/nod/dev/projects/iree/build_tools/pkgci/bisect/../setup_venv.py /home/nod/.iree/bisect/${REF_HASH}/.venv --artifact-path=/home/nod/.iree/bisect/${REF_HASH} --fetch-git-ref=${REF_HASH} +PATH="/home/nod/.iree/bisect/$REF_HASH/.venv/bin:$PATH" + +set +e + +######################################### +############ ORIGINAL SCRIPT ############ +######################################### + +iree-compile --iree-hal-target-backends=llvm-cpu -o /dev/null /home/nod/.iree/bisect/issue_18879.mlir + +######################################### +##### BISECT RELEASE SCRIPT CLEANUP ##### +######################################### + +RET_VALUE=$? +if [ $RET_VALUE -ne 0 ]; then + exit 1 +fi +``` + +### Example annotated logs + +Raw logs here: https://gist.github.com/ScottTodd/cff468a50df63b65e5c5f449fabab6af + +```bash +$ ./bisect_packages.py \ + --good-ref=candidate-20241016.1048 \ + --bad-ref=candidate-20241017.1049 \ + --test-script=/home/nod/.iree/bisect/issue_18879.sh + +Welcome to bisect_packages.py! + +------------------------------------------------------------------ +--------- Configuration ------------------------------------------ +------------------------------------------------------------------ + + Searching range : 'candidate-20241016.1048' - 'candidate-20241017.1049' + Using working directory : '/home/nod/.iree/bisect' + Using test script : '/home/nod/.iree/bisect/issue_18879.sh' + +------------------------------------------------------------------ + +------------------------------------------------------------------ +--------- Running git bisect ------------------------------------- +------------------------------------------------------------------ + +# -------------------------------------- +# Here we start to test the first commit +# -------------------------------------- + +Bisecting: 5 revisions left to test after this (roughly 3 steps) +[c7213deeb5c7abb0843088815580793b282fdc34] Produce releases for Python 3.13. (#18799) +running '/home/nod/.iree/bisect/bisect_run_script.sh' +++ git rev-parse BISECT_HEAD ++ REF_HASH=c7213deeb5c7abb0843088815580793b282fdc34 ++ /home/nod/dev/projects/iree/build_tools/pkgci/setup_venv_for_ref.py c7213deeb5c7abb0843088815580793b282fdc34 --work-dir /home/nod/.iree/bisect +------------------------------------------------------------------ +Installing packages for ref: c7213deeb5c7abb0843088815580793b282fdc34 + Using base working directory : '/home/nod/.iree/bisect' + +# ----------------------------------------------------- +# Here we download and install packages for that commit +# ----------------------------------------------------- +Running command to list workflow runs: + gh api -H Accept: application/vnd.github+json -H X-GitHub-Api-Version: 2022-11-28 /repos/iree-org/iree/actions/workflows/pkgci.yml/runs?head_sha=c7213deeb5c7abb0843088815580793b282fdc34 +Found workflow run: https://github.com/iree-org/iree/actions/runs/11375010806 +Found cached .whl files in artifacts dir, skipping download +Creating venv at '/home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/.venv' + +Running command to install dependencies: + /home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/.venv/bin/python -m pip install --quiet numpy sympy +Running command to install package: + /home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/.venv/bin/python -m pip install --quiet /home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/iree_compiler-0.dev1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl +Running command to install package: + /home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/.venv/bin/python -m pip install --quiet /home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/iree_runtime-0.dev1-cp311-cp311-manylinux_2_28_x86_64.whl + +Checking packages with 'pip freeze': +iree-compiler @ file:///home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/iree_compiler-0.dev1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl#sha256=4078073daae1b706361091389753a4887bfa7d4797ea66dce1d0daaa5bffc58c +iree-runtime @ file:///home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/iree_runtime-0.dev1-cp311-cp311-manylinux_2_28_x86_64.whl#sha256=564779699f560ba1da406c3d7d08fc75ba8b8eb2f6fc6e074e691a34bbb29bdf +mpmath==1.3.0 +numpy==2.1.3 +sympy==1.13.3 +------------------------------------------------------------------ + ++ PATH=/home/nod/.iree/bisect/c7213deeb5c7abb0843088815580793b282fdc34/.venv/bin:/usr/lib/git-core:/usr/lib/git-core:/home/nod/.pyenv/libexec:/home/nod/.pyenv/plugins/python-build/bin:/home/nod/.pyenv/plugins/pyenv-virtualenv/bin:/home/nod/.pyenv/plugins/pyenv-update/bin:/home/nod/.pyenv/plugins/pyenv-doctor/bin:/home/nod/.pyenv/shims:/home/nod/.pyenv/bin:/home/nod/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin +# ----------------------------------------------------- +# Here we run the test script +# ----------------------------------------------------- ++ set +e ++ iree-compile --iree-hal-target-backends=llvm-cpu -o /dev/null /home/nod/.iree/bisect/issue_18879.mlir +/home/nod/.iree/bisect/issue_18879.mlir:17:11: error: operand #0 does not dominate this use + %21 = torch.operator "onnx.Resize"(%20, %none, %1) {torch.onnx.coordinate_transformation_mode = "asymmetric", torch.onnx.cubic_coeff_a = -7.500000e-01 : f32, torch.onnx.mode = "nearest", torch.onnx.nearest_mode = "floor"} : (!torch.vtensor<[1,18,14,14],f32>, !torch.none, !torch.vtensor<[4],f32>) -> !torch.vtensor<[1,18,56,56],f32> + ^ +/home/nod/.iree/bisect/issue_18879.mlir:17:11: note: see current operation: %144 = "tensor.extract"(%32, %1, %129, %137, %143) : (tensor<1x18x14x14xf32>, index, index, index, index) -> f32 +/home/nod/.iree/bisect/issue_18879.mlir:16:11: note: operand defined here (op in a parent region) + %20 = torch.operator "onnx.Conv"(%arg2, %2, %3) {torch.onnx.dilations = [1 : si64, 1 : si64], torch.onnx.group = 1 : si64, torch.onnx.kernel_shape = [1 : si64, 1 : si64], torch.onnx.pads = [0 : si64, 0 : si64, 0 : si64, 0 : si64], torch.onnx.strides = [1 : si64, 1 : si64]} : (!torch.vtensor<[1,72,14,14],f32>, !torch.vtensor<[18,72,1,1],f32>, !torch.vtensor<[18],f32>) -> !torch.vtensor<[1,18,14,14],f32> + ^ ++ RET_VALUE=1 ++ '[' 1 -ne 0 ']' ++ exit 1 +# -------------------------------------------------------------------- +# The test script completed, so now we proceed to test the next commit +# -------------------------------------------------------------------- +Bisecting: 2 revisions left to test after this (roughly 2 steps) +[8568efa3cceb6dbbd69e8b681436a17efcce1a74] [GPU] Adding support for opt pass plugins during AMDGPU executable serialization (#18347) + +# -------------------------------------------------------------------- +# (repeat the download packages --> run test script step for other commits) +# ... skipping ahead ... +# -------------------------------------------------------------------- + +# ----------------------------------------- +# Bisecting finished. Here are the findings +# ----------------------------------------- +206b60ca59c9dbbca5769694df4714c38cecaced is the first bad commit +commit 206b60ca59c9dbbca5769694df4714c38cecaced +Author: Ian Wood <75152913+IanWood1@users.noreply.github.com> +Date: Wed Oct 16 10:52:47 2024 -0700 + + [DispatchCreation] Extend multi-use producer fusion (#18551) + + Fuse even in cases where the most dominant op isn't fusable, but other operations would be legal to fuse. Do this by moving the fusable consumer and all transitive defs before all other consumers (if legal). + + --------- + + Signed-off-by: Ian Wood + + .github/workflows/pkgci_regression_test.yml | 4 +- + .../FuseHorizontalContractions.cpp | 61 ++--------------- + .../FuseMultiUseElementwiseProducer.cpp | 76 +++++++++++++++++----- + .../iree/compiler/DispatchCreation/FusionUtils.cpp | 33 ++++++++++ + .../iree/compiler/DispatchCreation/FusionUtils.h | 44 +++++++++++++ + .../test/fuse_multiuse_elementwise_producer.mlir | 25 +++++++ + 6 files changed, 169 insertions(+), 74 deletions(-) +bisect found first bad commit +``` + +### Development notes + +Testing bisect: + +```bash +pyenv shell 3.11 + +./bisect_packages.py \ + --good-ref=iree-3.0.0 \ + --bad-ref=iree-3.1.0rc20241122 \ + --test-script=./bisect_example_timestamp.sh + +# 5b0740c97a33edce29e753b14b9ff04789afcc53 is the first bad commit +``` diff --git a/build_tools/pkgci/bisect/bisect_example_timestamp.sh b/build_tools/pkgci/bisect/bisect_example_timestamp.sh new file mode 100755 index 000000000000..d1b61a05f45e --- /dev/null +++ b/build_tools/pkgci/bisect/bisect_example_timestamp.sh @@ -0,0 +1,43 @@ +#!/bin/bash +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Simple example script to test bisect with. +# Fails for commits after Nov 19, 2024. + +# Example usage with https://git-scm.com/docs/git-bisect: +# +# git bisect start --no-checkout --first-parent +# git bisect good iree-3.0.0 +# git bisect bad iree-3.1.0rc20241122 +# git bisect run bisect_example_timestamp.sh +# +# running 'bisect_example_timestamp.sh' +# Commit 26ef79aa7c has timestamp: 1732059549 +# Timestamp >= 1732000000, exit 1 +# Bisecting: 10 revisions left to test after this (roughly 4 steps) +# ... +# 5b0740c97a33edce29e753b14b9ff04789afcc53 is the first bad commit +# +# Example usage with ./bisect_packages.py (even though this doesn't use any +# release artifacts like `iree-compile`): +# +# ./bisect_packages.py \ +# --good-ref=iree-3.0.0 \ +# --bad-ref=iree-3.1.0rc20241122 \ +# --test-script=./bisect_example_timestamp.sh + +SHORT_HASH=$(git rev-parse --short BISECT_HEAD) +COMMIT_TIMESTAMP=$(git show --no-patch --format=%ct BISECT_HEAD) +echo "Commit ${SHORT_HASH} has timestamp: ${COMMIT_TIMESTAMP}" + +if [ "$COMMIT_TIMESTAMP" -gt "1732000000" ]; then + echo " Timestamp >= 1732000000, exit 1" + exit 1 +else + echo " Timestamp < 1732000000, exit 0" + exit 0 +fi diff --git a/build_tools/pkgci/bisect/bisect_packages.py b/build_tools/pkgci/bisect/bisect_packages.py new file mode 100755 index 000000000000..33416dbb5f54 --- /dev/null +++ b/build_tools/pkgci/bisect/bisect_packages.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +"""Dev package bisect script. + +This connects the `git bisect` tool (https://git-scm.com/docs/git-bisect) +with IREE's package builds, allowing developers to run tests through commit +history efficiently. For example, this can be used to spot at which commit +an `iree-compile` command started failing. + +Requirements: + git (https://git-scm.com/) + gh (https://cli.github.com/) + Linux (at least until IREE builds packages for other systems at each commit) + Python 3.11 + +Example usage: + bisect_packages.py \ + --good-ref=iree-3.0.0 \ + --bad-ref=iree-3.1.0rc20241122 \ + --test-script=bisect_example_timestamp.sh +""" + + +import argparse +import os +import platform +import shutil +import subprocess +import sys +from pathlib import Path + +THIS_DIR = Path(__file__).parent.resolve() +REPO_ROOT = THIS_DIR.parent.parent.parent + + +def parse_arguments(): + parser = argparse.ArgumentParser(description="Git release bisect tool") + # TODO(scotttodd): add --interactive mode that prompts like git bisect does + parser.add_argument( + "--good-ref", + help="The git ref (commit hash, branch name, tag name, etc.) at the lower end of the range", + required=True, + ) + parser.add_argument( + "--bad-ref", + help="The git ref (commit hash, branch name, tag name, etc.) at the upper end of the range", + required=True, + ) + parser.add_argument( + "--work-dir", + help="The working directory to use. Defaults to ~/.iree/bisect/", + default=Path.home() / ".iree" / "bisect", + type=Path, + ) + # TODO(scotttodd): choice between manual or script (`git bisect run`) to use + # note that a "manual" mode would need developers to run + # ```bash + # REF_HASH=$(git rev-parse BISECT_HEAD) + # python3.11 setup_venv.py \ + # $WORKDIR/$REF_HASH/.venv \ + # --artifact-path=$WORKDIR/$REF_HASH \ + # --fetch-git-ref=$REF_HASH + # source $WORKDIR/$REF_HASH/.venv/bin/activate + # ``` + parser.add_argument( + "--test-script", + help="The script to run at each commit", + required=True, + ) + parser.add_argument( + "--ignore-system-requirements", + help="Ignores system requirements like Python 3.11 and tries to run even if they are not met.", + action="store_true", + default=False, + ) + # TODO(scotttodd): --clean arg to `rm -rf` the workdir + # TODO(scotttodd): control over logging + # redirect stdout/stderr from test script separate files in the workdir? + + return parser.parse_args() + + +def check_system_requirements(ignore_system_requirements): + print("") + system_check_okay = True + + # Check for Linux. + print( + f" Current platform is '{platform.platform()}', platform.system is '{platform.system()}'." + ) + if "Linux" not in platform.system(): + print(" ERROR! platform.system must be 'Linux'.", file=sys.stderr) + system_check_okay = False + + # Check for Python 3.11. + print("") + print(f" Current Python version is '{sys.version}'. This script requires 3.11.") + if sys.version_info[:2] == (3, 11): + python311_path = "python" + else: + python311_path = shutil.which("python3.11") + if python311_path: + print(f" Found python3.11 at '{python311_path}', using that instead.") + else: + print( + " ERROR! Could not find Python version 3.11. Python version must be 3.11 to match package builds.", + file=sys.stderr, + ) + print( + " See `.github/workflows/pkgci_build_packages.yml` and `build_tools/pkgci/build_linux_packages.sh`.", + file=sys.stderr, + ) + system_check_okay = False + + # Check for 'gh'. + print("") + gh_path = shutil.which("gh") + if not gh_path: + print( + " ERROR! Could not find 'gh'. Install by following https://github.com/cli/cli#installation.", + file=sys.stderr, + ) + system_check_okay = False + else: + print(f" Found gh at '{gh_path}'.") + + if not system_check_okay: + print("") + if ignore_system_requirements: + print( + "One or more configuration issues detected, but --ignore-system-requirements is set. Continuing.", + file=sys.stderr, + ) + return + print( + "One or more configuration issues detected. Fix the reported issues or pass --ignore-system-requirements to try running anyways. Exiting.", + file=sys.stderr, + ) + print("") + print("------------------------------------------------------------------") + sys.exit(1) + + return python311_path + + +def main(args): + print("Welcome to bisect_packages.py!") + + print("") + print("------------------------------------------------------------------") + print("--------- Configuration ------------------------------------------") + print("------------------------------------------------------------------") + print("") + print(f" Searching range : '{args.good_ref}' - '{args.bad_ref}'") + + print(f" Using working directory : '{args.work_dir}'") + Path.mkdir(args.work_dir, parents=True, exist_ok=True) + + print(f" Using test script : '{args.test_script}'") + + python311_path = check_system_requirements(args.ignore_system_requirements) + + print("") + print("------------------------------------------------------------------") + + # Create new script in working directory that: + # * downloads the packages from the release and installs them + # * runs the original test script + bisect_run_script = args.work_dir / "bisect_run_script.sh" + with open(bisect_run_script, "w") as bisect_run_script_file: + contents = "" + contents += "#!/bin/bash\n" + + contents += "\n" + contents += "#########################################\n" + contents += "###### BISECT RELEASE SCRIPT SETUP ######\n" + contents += "#########################################\n" + contents += "\n" + contents += "set -xeuo pipefail\n" + contents += "\n" + + # Download packages for REF_HASH and install them into REF_HASH/.venv/. + contents += "REF_HASH=$(git rev-parse BISECT_HEAD)\n" + contents += f'"{python311_path}" ' + contents += str((THIS_DIR / ".." / "setup_venv.py").as_posix()) + contents += f" {args.work_dir}/" + contents += "${REF_HASH}/.venv" + contents += f" --artifact-path={args.work_dir}/" + contents += "${REF_HASH} " + contents += " --fetch-git-ref=${REF_HASH}\n" + # Prepend the venv bin dir to $PATH. This is similar to running + # `source .venv/bin/activate` + # while scoped to this process. Note that this does not modify + # $PYTHONHOME or support the `deactivate` command. + contents += f'PATH="{args.work_dir}/$REF_HASH/.venv/bin:$PATH"\n' + + contents += "\n" + # Controlled failure - don't immediately exit. See below. + contents += "set +e\n" + contents += "\n" + contents += "#########################################\n" + contents += "############ ORIGINAL SCRIPT ############\n" + contents += "#########################################\n" + contents += "\n" + + with open(args.test_script, "r") as original_script: + contents += original_script.read() + + contents += "\n" + contents += "#########################################\n" + contents += "##### BISECT RELEASE SCRIPT CLEANUP #####\n" + contents += "#########################################\n" + contents += "\n" + # Controlled failure, See `set +e` above. + # `git bisect` is looking for exit values in the 1-127 range, while + # iree-compile can exit with value 245 sometimes: + # https://git-scm.com/docs/git-bisect#_bisect_run. Here we just check + # for non-zero and normalize back to 1. + contents += "RET_VALUE=$?\n" + contents += "if [ $RET_VALUE -ne 0 ]; then\n" + contents += " exit 1\n" + contents += "fi\n" + + bisect_run_script_file.write(contents) + + os.chmod(str(bisect_run_script), 0o744) # Set as executable. + + print("") + print("------------------------------------------------------------------") + print("--------- Running git bisect -------------------------------------") + print("------------------------------------------------------------------") + print("") + subprocess.check_call(["git", "bisect", "reset"], cwd=REPO_ROOT) + subprocess.check_call( + [ + "git", + "bisect", + "start", + # Just update the BISECT_HEAD reference instead of checking out the + # ref for each iteration of the bisect process. We won't be building + # from source and this script lives in the source tree, so keep the + # repository in a stable state. + # Note: scripts can access the hash via `git rev-parse BISECT_HEAD`. + "--no-checkout", + # We only care about the merge/aggregate commit when branches were + # merged. Ignore ancestors of merge commits. + "--first-parent", + ], + cwd=REPO_ROOT, + ) + subprocess.check_call(["git", "bisect", "good", args.good_ref], cwd=REPO_ROOT) + subprocess.check_call(["git", "bisect", "bad", args.bad_ref], cwd=REPO_ROOT) + subprocess.check_call( + ["git", "bisect", "run", str(bisect_run_script)], cwd=REPO_ROOT + ) + + print("") + + +if __name__ == "__main__": + main(parse_arguments()) diff --git a/build_tools/pkgci/setup_venv.py b/build_tools/pkgci/setup_venv.py index d0d51638981f..8034edbb6374 100755 --- a/build_tools/pkgci/setup_venv.py +++ b/build_tools/pkgci/setup_venv.py @@ -7,14 +7,58 @@ """Sets up a Python venv with compiler/runtime from a workflow run. -There are two modes in which to use this script: +There are several modes in which to use this script: -* Within a workflow, an artifact action will typically be used to fetch - relevant package artifacts. Specify the fetch location with - `--artifact-path=`. +* Within a workflow triggered by `workflow_call`, an artifact action will + typically be used to fetch relevant package artifacts. Specify the fetched + location with `--artifact-path=`: -* Locally, the `--fetch-gh-workflow=WORKFLOW_ID` can be used instead in order - to download and setup the venv in one step. + ```yml + - uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} + ``` + +* Within a workflow triggered by `workflow_dispatch`, pass `artifact_run_id` as + an input that developers must specify when running the workflow: + + ```yml + on: + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + + ... + steps: + - name: Setup venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + ``` + + (Note that these two modes are often combined to allow for workflow testing) + +* Locally, the `--fetch-gh-workflow=WORKFLOW_ID` can be used to download and + setup the venv from a specific workflow run in one step: + + + ```bash + python3.11 ./build_tools/pkgci/setup_venv.py /tmp/.venv --fetch-gh-workflow=11977414405 + ``` + +* Locally, the `--fetch-git-ref=GIT_REF` can be used to download and setup the + venv from the latest workflow run for a given ref (commit) in one step: + + ```bash + python3.11 ./build_tools/pkgci/setup_venv.py /tmp/.venv --fetch-git-ref=main + ``` You must have the `gh` command line tool installed and authenticated if you will be fetching artifacts. @@ -34,10 +78,70 @@ import tempfile import zipfile +THIS_DIR = Path(__file__).parent.resolve() +REPO_ROOT = THIS_DIR.parent.parent + + +def parse_arguments(argv=None): + parser = argparse.ArgumentParser(description="Setup venv") + parser.add_argument( + "venv_dir", type=Path, help="Directory in which to create the venv" + ) + parser.add_argument("--artifact-path", help="Path in which to find/fetch artifacts") + + fetch_group = parser.add_mutually_exclusive_group() + fetch_group.add_argument( + "--fetch-gh-workflow", help="Fetch artifacts from a GitHub workflow" + ) + fetch_group.add_argument("--fetch-git-ref", help="Fetch artifacts for a git ref") + + parser.add_argument( + "--compiler-variant", + default="", + help="Package variant to install for the compiler ('', 'asserts')", + ) + parser.add_argument( + "--runtime-variant", + default="", + help="Package variant to install for the runtime ('', 'asserts')", + ) + args = parser.parse_args(argv) + return args + + +def get_latest_workflow_run_id_for_ref(ref: str) -> int: + print(f"Normalizing ref: {ref}") + normalized_ref = ( + subprocess.check_output(["git", "rev-parse", ref], cwd=REPO_ROOT) + .decode() + .strip() + ) + + print(f"Fetching artifacts for normalized ref: {normalized_ref}") + base_path = f"/repos/iree-org/iree" + workflow_run_args = [ + "gh", + "api", + "-H", + "Accept: application/vnd.github+json", + "-H", + "X-GitHub-Api-Version: 2022-11-28", + f"{base_path}/actions/workflows/pkgci.yml/runs?head_sha={normalized_ref}", + ] + print(f"Running command to list workflow runs:\n {' '.join(workflow_run_args)}") + workflow_run_output = subprocess.check_output(workflow_run_args) + workflow_run_json_output = json.loads(workflow_run_output) + if workflow_run_json_output["total_count"] == 0: + raise RuntimeError("Workflow did not run at this commit") + + latest_run = workflow_run_json_output["workflow_runs"][-1] + print(f"Found workflow run: {latest_run['html_url']}") + return latest_run["id"] + @functools.lru_cache def list_gh_artifacts(run_id: str) -> Dict[str, str]: - print(f"Fetching artifacts for workflow run {run_id}") + print(f"Fetching artifacts for workflow run: {run_id}") base_path = f"/repos/iree-org/iree" output = subprocess.check_output( [ @@ -87,30 +191,14 @@ def find_venv_python(venv_path: Path) -> Optional[Path]: return None -def parse_arguments(argv=None): - parser = argparse.ArgumentParser(description="Setup venv") - parser.add_argument("--artifact-path", help="Path in which to find/fetch artifacts") - parser.add_argument( - "--fetch-gh-workflow", help="Fetch artifacts from a GitHub workflow" - ) - parser.add_argument( - "--compiler-variant", - default="", - help="Package variant to install for the compiler ('', 'asserts')", - ) - parser.add_argument( - "--runtime-variant", - default="", - help="Package variant to install for the runtime ('', 'asserts')", - ) - parser.add_argument( - "venv_dir", type=Path, help="Directory in which to create the venv" - ) - args = parser.parse_args(argv) - return args - - def main(args): + # Look up the workflow run for a ref. + if args.fetch_git_ref: + latest_gh_workflow = get_latest_workflow_run_id_for_ref(args.fetch_git_ref) + args.fetch_git_ref = "" + args.fetch_gh_workflow = str(latest_gh_workflow) + return main(args) + # Make sure we have an artifact path if fetching. if not args.artifact_path and args.fetch_gh_workflow: with tempfile.TemporaryDirectory() as td: diff --git a/docs/website/docs/developers/debugging/compile-time-regressions.md b/docs/website/docs/developers/debugging/compile-time-regressions.md index f8d7a04900e6..76c7f40f5359 100644 --- a/docs/website/docs/developers/debugging/compile-time-regressions.md +++ b/docs/website/docs/developers/debugging/compile-time-regressions.md @@ -48,6 +48,12 @@ Building the compiler from source and using specific commits in IREE, though it typically won't let you step through changes in submodules (e.g. MLIR updates in `third_party/llvm-project/`). +#### Scripted bisecting with package artifacts + +See . + +#### Manually bisecting with source builds + **Tip**: [Configure ccache](../building/cmake-with-ccache.md) if you'll be rebuilding the compiler while bisecting @@ -71,8 +77,7 @@ git bisect bad [] An automated workflow can use `git bisect run` and a script: -```shell -# run_bisect.sh +```shell title="run_bisect.sh" git submodule update cmake --build build/ --target iree-compile # Other logic here @@ -87,7 +92,7 @@ git bisect run run_bisect.sh #### Sample: compile executable sources individually with a timeout -```bash +```bash title="run_bisect.sh" #!/bin/bash set -xeuo pipefail