Skip to content

Commit

Permalink
print sccache stats in builds (#413)
Browse files Browse the repository at this point in the history
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`
* modifying `dependencies.yaml` to match RAPIDS-wide naming conventions

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #413
  • Loading branch information
jameslamb authored Oct 29, 2024
1 parent 9f035d8 commit d296d81
Show file tree
Hide file tree
Showing 5 changed files with 33 additions and 9 deletions.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ repos:
cpp/cmake/modules/FindAVX\.cmake|
- id: verify-alpha-spec
- repo: https://github.com/rapidsai/dependency-file-generator
rev: v1.13.11
rev: v1.16.0
hooks:
- id: rapids-dependency-file-generator
args: ["--clean"]
Expand Down
4 changes: 4 additions & 0 deletions ci/build_cpp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ rapids-print-env

rapids-logger "Begin cpp build"

sccache --zero-stats

RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry mambabuild conda/recipes/libcuvs

sccache --show-adv-stats

rapids-upload-conda-to-s3 cpp
10 changes: 10 additions & 0 deletions ci/build_python.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,20 +24,28 @@ version=$(rapids-generate-version)
export RAPIDS_PACKAGE_VERSION=${version}
echo "${version}" > VERSION

sccache --zero-stats

# TODO: Remove `--no-test` flags once importing on a CPU
# node works correctly
rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
conda/recipes/cuvs

sccache --show-adv-stats
sccache --zero-stats

# Build cuvs-bench for each cuda and python version
rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
--channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \
conda/recipes/cuvs-bench

sccache --show-adv-stats
sccache --zero-stats

# Build cuvs-bench-cpu only in CUDA 12 jobs since it only depends on python
# version
RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}"
Expand All @@ -47,6 +55,8 @@ if [[ ${RAPIDS_CUDA_MAJOR} == "12" ]]; then
--channel "${CPP_CHANNEL}" \
--channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \
conda/recipes/cuvs-bench-cpu

sccache --show-adv-stats
fi

rapids-upload-conda-to-s3 python
16 changes: 13 additions & 3 deletions ci/build_wheel.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,20 @@ case "${RAPIDS_CUDA_VERSION}" in
;;
esac

# Hardcode the output dir
python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check
rapids-logger "Building '${package_name}' wheel"

sccache --zero-stats

python -m pip wheel \
-w dist \
-v \
--no-deps \
--disable-pip-version-check \
.

sccache --show-adv-stats

mkdir -p final_dist
python -m auditwheel repair -w final_dist "${EXCLUDE_ARGS[@]}" dist/*

RAPIDS_PY_WHEEL_NAME="${underscore_package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 final_dist
RAPIDS_PY_WHEEL_NAME="${underscore_package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 python final_dist
10 changes: 5 additions & 5 deletions dependencies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,14 @@ files:
- rapids_build
- cuda
- rust
py_build_py_cuvs:
py_build_cuvs:
output: pyproject
pyproject_dir: python/cuvs
extras:
table: build-system
includes:
- build
py_rapids_build_py_cuvs:
py_rapids_build_cuvs:
output: pyproject
pyproject_dir: python/cuvs
extras:
Expand All @@ -90,7 +90,7 @@ files:
includes:
- rapids_build
- build_py_cuvs
py_run_py_cuvs:
py_run_cuvs:
output: pyproject
pyproject_dir: python/cuvs
extras:
Expand All @@ -99,7 +99,7 @@ files:
- cuda_wheels
- run_py_cuvs
- depends_on_pylibraft
py_test_py_cuvs:
py_test_cuvs:
output: pyproject
pyproject_dir: python/cuvs
extras:
Expand All @@ -116,7 +116,7 @@ files:
table: build-system
includes:
- rapids_build_setuptools
py_rapids_build_py_cuvs_bench:
py_rapids_build_cuvs_bench:
output: pyproject
pyproject_dir: python/cuvs_bench
extras:
Expand Down

0 comments on commit d296d81

Please sign in to comment.