Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new conda environments #120

Open
wants to merge 2 commits into
base: branch-0.35
Choose a base branch
from
Open

Add new conda environments #120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
65c84b7
Add conda environment files without CUDA requirement
pentschev Nov 7, 2023
af5b587
Update README
pentschev Nov 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,12 @@ $ conda install -c conda-forge mamba
After that, one can proceed as before, but simply replacing `conda` with `mamba` in the environment creation command:

```
$ mamba env create -n ucxx -f conda/environments/ucxx-cuda118_arch-x86_64.yml
$ conda activate ucxx
$ mamba env create -n ucxx -f conda/environments/all_cuda-118_arch-x86_64.yaml
$ conda activate all_cuda-118_arch-x86_64
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should match the name from -n ucxx above.

Suggested change
$ conda activate all_cuda-118_arch-x86_64
$ conda activate ucxx

```

The environment above will install CUDA 11.8 dependencies, with another file `all_cuda-120_arch-x86_64.yaml` for CUDA 12.0. There are also two other files if you don't need CUDA: `cpp.yaml` (C++-only build dependencies) and `all.yaml` (C++ and Python build dependencies).

### Convenience Script

For convenience, we provide the `./build.sh` script. By default, it will build and install both C++ and Python libraries. For a detailed description on available options please check `./build.sh --help`.
Expand Down
34 changes: 34 additions & 0 deletions conda/environments/all.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# This file is generated by `rapids-dependency-file-generator`.
Copy link
Contributor

@bdice bdice Nov 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All environment files need to be generated by dependencies.yaml.

The failure in pre-commit is because rapids-dependency-file-generator looks for files that it generated and removes them before re-generating the new files. You have the header here saying it's a generated file, but there's no declaration of this environment file in dependencies.yaml.

This is how you tell it to generate a given environment file:

ucxx/dependencies.yaml

Lines 3 to 17 in 4464ca9

all:
output: conda
matrix:
cuda: ["11.8", "12.0"]
arch: [x86_64]
includes:
- build_cpp
- build_python
- checks
- cudatoolkit
- dev
- py_version
- run_python
- test_cpp
- test_python

For example, you might want to add a section like this for the C++ dependency list.

files:
  cpp:     # This is used as part of the environment's filename
    output: conda
    matrix:
      arch: [x86_64]
    includes:
      - build_cpp
      - checks
      - dev
      - test_cpp

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Bradley, this is helpful. I'd still like to trim that more though, for example the above will result in:

# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- autoconf
- automake
- c-compiler
- cmake>=3.26.4
- cxx-compiler
- dask-cuda==23.12.*
- dask-cudf==23.12.*
- fmt>=9.1.0,<10
- gmock>=1.13.0
- gtest>=1.13.0
- librmm==23.12.*
- libtool
- ninja
- pip
- pkg-config
- pre-commit
- spdlog>=1.11.0,<1.12
name: cpp_arch-x86_64

That include a bunch of things we don't need: rapidsai/rapidsai-nightly/dask/label/dev/nvidia channels and dask-cuda/dask-cudf/librmm/pip/pkg-config. Is there a way to minimize that?

Additionally, this is supposed to be valid for aarch64 too, can we make it architecture agnostic?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's currently no way to drop extra channels from the generated files. You can file an issue to rapids-dependency-file-generator if you have a proposal for how the dependencies.yaml schema should accommodate that idea, but I'm not sure if it's worth changing. https://github.com/rapidsai/dependency-file-generator

If you depend only on architecture-agnostic packages, you can just drop the arch from the files: spec.

files:
  cpp:     # This is used as part of the environment's filename
    output: conda
    includes:
      - build_cpp
      - checks
      - dev
      - test_cpp

# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- dask/label/dev
- conda-forge
dependencies:
- c-compiler
- cloudpickle
- cmake>=3.26.4
- cxx-compiler
- cython>=3.0.0
- dask
- distributed
- fmt>=9.1.0,<10
- gmock>=1.13.0
- gtest>=1.13.0
- libtool
- ninja
- numba>=0.57.1
- numpy>=1.21
- pip
- pkg-config
- pre-commit
- pytest
- pytest-asyncio
- pytest-rerunfailures
- python>=3.9,<3.11
- scikit-build>=0.13.1
- setuptools
- spdlog>=1.11.0,<1.12
- tomli
- ucx
- wheel
name: ucxx
17 changes: 17 additions & 0 deletions conda/environments/cpp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- conda-forge
dependencies:
- c-compiler
- cmake>=3.26.4
- cxx-compiler
- fmt>=9.1.0,<10
- gmock>=1.13.0
- gtest>=1.13.0
- libtool
- ninja
- pre-commit
- spdlog>=1.11.0,<1.12
- ucx
name: libucxx
Loading