Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for python 3.11 #176

Merged
merged 10 commits into from
Feb 26, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/conda-python-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,14 @@ jobs:
- { CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.9' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.10' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.10' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.11' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.11' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.9' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.9' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.10' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.10' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.11' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.11' }
"

echo "MATRIX=$(
Expand All @@ -73,6 +77,7 @@ jobs:
fail-fast: false
matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
runs-on: "linux-${{ matrix.ARCH }}-${{ inputs.node_type }}"
continue-on-error: ${{ matrix.PY_VER == '3.11' }}
jameslamb marked this conversation as resolved.
Show resolved Hide resolved
env:
RAPIDS_ARTIFACTS_DIR: ${{ github.workspace }}/artifacts
container:
Expand Down
9 changes: 5 additions & 4 deletions .github/workflows/conda-python-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,16 +69,16 @@ jobs:
- { CUDA_VER: '11.4.3', LINUX_VER: 'centos7', ARCH: 'amd64', PY_VER: '3.9', GPU: 'v100', DRIVER: 'earliest' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.10', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.10', GPU: 'a100', DRIVER: 'latest' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.10', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.11', GPU: 'v100', DRIVER: 'latest' }
nightly:
- { CUDA_VER: '11.4.3', LINUX_VER: 'centos7', ARCH: 'amd64', PY_VER: '3.9', GPU: 'v100', DRIVER: 'earliest' }
- { CUDA_VER: '11.4.3', LINUX_VER: 'ubuntu20.04', ARCH: 'amd64', PY_VER: '3.9', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '11.4.3', LINUX_VER: 'ubuntu20.04', ARCH: 'arm64', PY_VER: '3.9', GPU: 'a100', DRIVER: 'latest' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.10', GPU: 'a100', DRIVER: 'latest' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', ARCH: 'amd64', PY_VER: '3.10', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', ARCH: 'amd64', PY_VER: '3.11', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.10', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.10', GPU: 'a100', DRIVER: 'latest' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.10', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', ARCH: 'arm64', PY_VER: '3.11', GPU: 'a100', DRIVER: 'latest' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', ARCH: 'amd64', PY_VER: '3.11', GPU: 'v100', DRIVER: 'latest' }
- { CUDA_VER: '12.2.2', LINUX_VER: 'rockylinux8', ARCH: 'arm64', PY_VER: '3.10', GPU: 'a100', DRIVER: 'latest' }
"

Expand All @@ -95,6 +95,7 @@ jobs:
fail-fast: false
matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
runs-on: "linux-${{ matrix.ARCH }}-gpu-${{ matrix.GPU }}-${{ matrix.DRIVER }}-1"
continue-on-error: ${{ matrix.PY_VER == '3.11' }}
env:
RAPIDS_COVERAGE_DIR: ${{ github.workspace }}/coverage-results
RAPIDS_TESTS_DIR: ${{ github.workspace }}/test-results
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/wheels-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,25 @@ jobs:
export MATRIX="
- { CUDA_VER: '11.8.0', ARCH: 'amd64', PY_VER: '3.9', LINUX_VER: 'centos7' }
- { CUDA_VER: '11.8.0', ARCH: 'amd64', PY_VER: '3.10', LINUX_VER: 'centos7' }
- { CUDA_VER: '11.8.0', ARCH: 'amd64', PY_VER: '3.11', LINUX_VER: 'centos7' }
- { CUDA_VER: '11.8.0', ARCH: 'arm64', PY_VER: '3.9', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '11.8.0', ARCH: 'arm64', PY_VER: '3.10', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '11.8.0', ARCH: 'arm64', PY_VER: '3.11', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '12.2.2', ARCH: 'amd64', PY_VER: '3.9', LINUX_VER: 'centos7' }
- { CUDA_VER: '12.2.2', ARCH: 'amd64', PY_VER: '3.10', LINUX_VER: 'centos7' }
- { CUDA_VER: '12.2.2', ARCH: 'amd64', PY_VER: '3.11', LINUX_VER: 'centos7' }
- { CUDA_VER: '12.2.2', ARCH: 'arm64', PY_VER: '3.9', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '12.2.2', ARCH: 'arm64', PY_VER: '3.10', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '12.2.2', ARCH: 'arm64', PY_VER: '3.11', LINUX_VER: 'rockylinux8' }
"

export NEW_MANYLINUX_MATRIX="
- { CUDA_VER: '11.8.0', ARCH: 'amd64', PY_VER: '3.9', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '11.8.0', ARCH: 'amd64', PY_VER: '3.10', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '11.8.0', ARCH: 'amd64', PY_VER: '3.11', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '12.2.2', ARCH: 'amd64', PY_VER: '3.9', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '12.2.2', ARCH: 'amd64', PY_VER: '3.10', LINUX_VER: 'rockylinux8' }
- { CUDA_VER: '12.2.2', ARCH: 'amd64', PY_VER: '3.11', LINUX_VER: 'rockylinux8' }
"

if ${{ inputs.build-2_28-wheels == 'true' }}; then
Expand All @@ -115,6 +121,8 @@ jobs:
strategy:
matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
runs-on: "linux-${{ matrix.ARCH }}-${{ inputs.node_type }}"
# TODO: remove 'continue-on-error' before closing https://github.com/rapidsai/build-planning/issues/3
continue-on-error: ${{ matrix.PY_VER == '3.11' }}
env:
RAPIDS_ARTIFACTS_DIR: ${{ github.workspace }}/artifacts
container:
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/wheels-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -71,19 +71,19 @@ jobs:
export MATRICES="
pull-request:
- { ARCH: 'amd64', PY_VER: '3.9', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu18.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu20.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', gpu: 'a100', driver: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu20.04', gpu: 'v100', driver: 'latest' }
nightly:
- { ARCH: 'amd64', PY_VER: '3.9', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu18.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu18.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.9', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', gpu: 'a100', driver: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', gpu: 'a100', driver: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', gpu: 'a100', driver: 'latest' }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s reshuffle these a bit. I want one configuration to be amd64 + latest Python + latest CUDA. Currently it seems only ARM has Python 3.11 jobs. I would also tilt the distribution of versions to be evenly weighted, it seems like the matrix tests 3.9 and 3.10 more than 3.11 (I didn’t do an exact count).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking @bdice .

This PR does currently have a Python 3.11 AMD job for wheels-test... but just on PRs.

- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu20.04', gpu: 'v100', driver: 'latest' }

Definitely agree the mix across Python versions is uneven right now, I'll fix that.

Before I go further with this... would you consider this change I just put up?

#177

Based on my experience here and in #166, I think it's worth taking a minute to re-sorting these matrices so they're easier to review.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright @bdice whenever you have time, could you please take another look?

(and @msarahan @AyodeAwe @KyleFromNVIDIA if you'd like another chance to review now that changes have been made)

By my count, on the current state of this branch we have the following mix:

workflow python version amd64 arm64 total
conda-python-build 3.9 2 2 4
conda-python-build 3.10 2 2 4
conda-python-build 3.11 2 2 4
conda-python-tests (pr) 3.9 1 0 1
conda-python-tests (pr) 3.10 1 1 2
conda-python-tests (pr) 3.11 1 0 1
conda-python-tests (nightly) 3.9 2 1 3
conda-python-tests (nightly) 3.10 2 1 3
conda-python-tests (nightly) 3.11 1 2 3
wheel-build 3.9 4 2 6
wheel-build 3.10 4 2 6
wheel-build 3.11 4 2 6
wheel-test (pr) 3.9 1 0 1
wheel-test (pr) 3.10 0 1 1
wheel-test (pr) 3.11 1 0 1
wheel-test (nightly) 3.9 2 1 3
wheel-test (nightly) 3.10 2 2 4
wheel-test (nightly) 3.11 2 1 3

And the conda-python-tests (pr) and wheel-test (pr) both have the combination "amd64 + latest Python + latest CUDA".

I've still preserved the same absolute number of test jobs as branch-24.04.

I've also updated the description, with tests from these two PRs:

- { ARCH: 'amd64', PY_VER: '3.9', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu18.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.9', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', gpu: 'a100', driver: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.9', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu20.04', gpu: 'v100', driver: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', gpu: 'a100', driver: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', gpu: 'a100', driver: 'latest' }
"

TEST_MATRIX=$(yq -n 'env(MATRICES) | .[strenv(BUILD_TYPE)]')
Expand All @@ -103,6 +103,8 @@ jobs:
strategy:
matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
runs-on: "linux-${{ matrix.ARCH }}-gpu-${{ matrix.gpu }}-${{ matrix.driver }}-1"
# TODO: remove 'continue-on-error' before closing https://github.com/rapidsai/build-planning/issues/3
continue-on-error: ${{ matrix.PY_VER == '3.11' }}
container:
image: "rapidsai/citestwheel:cuda${{ matrix.CUDA_VER }}-${{ matrix.LINUX_VER }}-py${{ matrix.PY_VER }}"
options: ${{ inputs.container-options }}
Expand Down