* Add ucx-py dependency to CI

…ns (rapidsai#1270)

Currently dask versions are pinned as part of every release cycle and then unpinned for the next development cycle across all of RAPIDS. This introduces a great deal of churn. To centralize the dependency, we have created a metapackage to manage the required dask version and this PR introduces that metapackage as a dependency of dask-cuda.

xref: rapidsai/cudf#14364

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Jake Awe (https://github.com/AyodeAwe)

URL: rapidsai#1270

Without this extra spec, consumers of dask-cuda nightlies won't know that dask-cuda nightlies want to use nightlies of rapids-dask-dependency.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)

URL: rapidsai#1273

The `package_name` variable was not set here, so the git commit was never actually overwritten.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1277

Closes rapidsai#1271

Authors:
  - Jacob Tomlinson (https://github.com/jacobtomlinson)
  - Lawrence Mitchell (https://github.com/wence-)

Approvers:
  - Benjamin Zaitlen (https://github.com/quasiben)

URL: rapidsai#1272

…apidsai#1289)

Increase Nanny close timeout for `test_spilling_local_cuda_cluster` which didn't fail in the past but failed for the first time in latest nightly run.

Authors:
   - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
   - Richard (Rick) Zamora (https://github.com/rjzamora)

RAPIDS repos are using the `main` branch of https://github.com/actions/labeler which recently introduced [breaking changes](https://github.com/actions/labeler/releases/tag/v5.0.0).

This PR pins to the latest v4 release of the labeler action until we can evaluate the changes required for v5.

Authors:
   - Ray Douglass (https://github.com/raydouglass)

Approvers:
   - AJ Schmidt (https://github.com/ajschmidt8)

I tried running update-version.sh and it properly updated the version, so my guess is that there was some race condition between when the script was updated and when it was run causing the version in pyproject.toml to not be properly updated.

Authors:
   - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
   - Bradley Dice (https://github.com/bdice)
   - Ray Douglass (https://github.com/raydouglass)

In rapidsai#1275 I erroneously assumed that updating dependencies.yaml would update pyproject.toml which is not currently the case, resulting in a mismatch between the two and the incorrect rapids-dask-dependency continuing to exist. This PR updates the repo to use dfg to generate pyproject.toml from dependencies.yaml. This PR also adds the pre-commit hook so that these two are automatically kept up-to-date going forward.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1276

I believe this is what is causing issues for downstream package resolution (24.02 != 24.2) e.g. in rapidsai/cudf#14426.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1279

Now that `rapids-dependency-file-generator` makes it easier to update environment files through a centralized config, does it make sense to commit these generated files directly to the repo?

Authors:
  - Charles Blackmon-Luca (https://github.com/charlesbluca)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1291

Nightly wheels also require `rapids-dask-dependency` which is only available in NVIDIA's PyPI index and cannot be published to PyPI as it installs Dask/Distributed from GitHub, which is forbidden by PyPI. Therefore, we're switching to publishing nightlies only to NVIDIA index as it doesn't seem external projects currently rely on nightlies. Release packages will continue to be published to PyPI.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Charles Blackmon-Luca (https://github.com/charlesbluca)
  - Ray Douglass (https://github.com/raydouglass)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: rapidsai#1294

Add timeout to `test_dask_use_explicit_comms` with SIGINT (i.e., KeyboardInterrupt) hoping that we can get a stacktrace that can help identifying the cause of the test deadlock in CI.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)

URL: rapidsai#1298

dask/dask#10738 has deprecated the `shuffle` kwarg in favor of `shuffle_method` which now raises a `FutureWarning`. This change transitions to the new kwarg.

Requires rapidsai/cudf#14708 .

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Richard (Rick) Zamora (https://github.com/rjzamora)

URL: rapidsai#1300

Double initialization of UCX context may raise exceptions and cause test failures, prevent that by reseting the context after doing some initial checks.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Benjamin Zaitlen (https://github.com/quasiben)

URL: rapidsai#1301

Contributes to rapidsai/build-planning#7.

Proposes splitting the `cuda-version` dependency in `dependencies.yaml` out to its own thing, separate from the bits of the CUDA Toolkit this project needs.

### Benefits of this change

* prevents accidental inclusion of multiple `cuda-version` version in environments
* reduces update effort (via enabling more use of globs like `"12.*"`)
* improves the chance that errors like "`conda` recipe is missing a dependency" are caught in CI

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1303

Reference: https://github.com/rapidsai/ops/issues/2766

Replace rapids-env-update with rapids-configure-conda-channels,
rapids-configure-sccache, and rapids-date-string.

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: rapidsai#1304

A recent change to the way `StringColumn`s are implemented in cudf threw up that we were never correctly determining the number of device buffers belonging to cudf columns if they had children (e.g. list and struct columns) or masks (any nullable column). Handle those cases and update the test.

Authors:
  - Lawrence Mitchell (https://github.com/wence-)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Mads R. B. Kristensen (https://github.com/madsbk)

URL: rapidsai#1305

dask-cuda uses pandas for some tests, but the main reason for the pinning is that it is inherited from RAPIDS libraries (mainly cudf) that do not yet support pandas 2.0 and are the primary use case for dask-cuda. However, there is no reason dask-cuda cannot be used in other contexts, so relaxing this constraint makes sense.

Resolves rapidsai#1306

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - Lawrence Mitchell (https://github.com/wence-)
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1308

Contributes to rapidsai/build-planning#13.

Updates `update-version.sh` to correctly handle RAPIDS dependencies like `cudf-cu12==24.2.*`.

This project doesn't appear to have any of those right now, but might in the future.

### How I tested this

The portability of this updated `sed` command was tested here: rapidsai/cudf#14825 (comment).

In this repo, I ran the following:

```shell
./ci/release/update-version.sh '23.10.00'
git diff

./ci/release/update-version.sh '24.04.00'
git diff
```

Confirmed that that first `git diff` changed all the things I expected, and that second one showed 0 changes.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1307

* switches to CUDA 12.2.2 for building conda packages and wheels
* adds new tests running against CUDA 12.2.2

This is part of ongoing work to build and test wheels against CUDA 12.2.2 across all of RAPIDS. For more details see:

* rapidsai/build-planning#7
* rapidsai/shared-workflows#166

### Notes for Reviewers

Planning a second round of PRs to revert these references back to a proper `branch-24.{nn}` release branch of `shared-workflows` once rapidsai/shared-workflows#166 is merged.

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)
  - Bradley Dice (https://github.com/bdice)

URL: rapidsai#1302

Dask CUDA must use the deprecated `dask.dataframe` API until rapidsai#1311 and rapidsai/cudf#15027 are both closed. This means that we must explicitly filter the following deprecation warning to avoid nighlty CI failures:

```
DeprecationWarning: The current Dask DataFrame implementation is deprecated. 
In a future release, Dask DataFrame will use new implementation that
contains several improvements including a logical query planning.
The user-facing DataFrame API will remain unchanged.

The new implementation is already available and can be enabled by
installing the dask-expr library:

    $ pip install dask-expr

and turning the query planning option on:

    >>> import dask
    >>> dask.config.set({'dataframe.query-planning': True})
    >>> import dask.dataframe as dd

API documentation for the new implementation is available at
https://docs.dask.org/en/stable/dask-expr-api.html

Any feedback can be reported on the Dask issue tracker
https://github.com/dask/dask/issues 

  import dask.dataframe as dd
```

This PR adds the (temporarily) necessary warning filter.

Authors:
  - Richard (Rick) Zamora (https://github.com/rjzamora)
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: rapidsai#1312

Follow-up to rapidsai#1302

For all GitHub Actions configs, replaces uses of the `test-cuda-12.2` branch on `shared-workflows`
with `branch-24.04`, now that rapidsai/shared-workflows#166 has been merged.

### Notes for Reviewers

This is part of ongoing work to build and test packages against CUDA 12.2 across all of RAPIDS.

For more details see:

* rapidsai/build-planning#7

*(created with `rapids-reviser`)*

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1314

Contributes to rapidsai/build-planning#3

This PR adds support for Python 3.11.

## Notes for Reviewers

This is part of ongoing work to add Python 3.11 support across RAPIDS.

The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged.

This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11.

See rapidsai/shared-workflows#176 for more details.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Jake Awe (https://github.com/AyodeAwe)

URL: rapidsai#1315

As NumPy 1.23 is needed for Python 3.11 support, go ahead and bump the minimum NumPy version used by Dask-CUDA to match that.

xref: rapidsai#1315
xref: rapidsai/build-planning#3

Authors:
  - https://github.com/jakirkham
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Jake Awe (https://github.com/AyodeAwe)

URL: rapidsai#1316

To eliminate hard-coding, generalize the GHA workflow logic to select one build for testing. This should simplify future Dask-CUDA updates.

xref: rapidsai/build-planning#25

Authors:
  - https://github.com/jakirkham

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1318

NumPy 2 is expected to be released in the near future. For the RAPIDS 24.04 release, we will pin to `numpy>=1.23,<2.0a0`. This PR adds an upper bound to affected RAPIDS repositories.

xref: rapidsai/build-planning#29

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: rapidsai#1320