Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build and ship C++ RAPIDS binaries #45

Open
vyasr opened this issue Apr 19, 2024 · 1 comment
Open

Build and ship C++ RAPIDS binaries #45

vyasr opened this issue Apr 19, 2024 · 1 comment

Comments

@vyasr
Copy link
Contributor

vyasr commented Apr 19, 2024

Currently the only way to access RAPIDS C++ libraries is via conda. There is no easy way to install RAPIDS C++ libraries in any other context. With #33, that will change to an extent since it will be possible to pip install the binaries. However, that is a very nontraditional approach for providing native libraries, and should not be our primary avenue for producing said binaries. However, the changes in #33 are also essentially a PoC that we can build the C++ libraries in our wheel containers and produce binaries that are portable, at least up to the requirements of the Python manylinux standard.

Once #33 is completed, we should take this one step further and build the C++ libraries standalone. We should be able to extract most of what we need directly from the C++ wheel building scripts since all native dependencies are already preinstalled into the wheel images and we know that these images produce fairly portable binaries. We can use CPack to produce native packages for whatever targets that we care about here. Then, the C++ wheel builds can be modified to simply include the entire contents of the CPacked package into the wheel install.

Getting this working will probably require some significant experimentation. Some notes:

  • I don't think a simple install(IMPORTED_RUNTIME_ARTIFACTS) of the CPacked library into the wheel will be sufficient because that will only install the library. What we want is to copy over everything required to make this a valid CMake packages, such as the CMake config files, as well as all the headers and anything else needed to compile against this package, in addition to the compiled libraries. However, we also may not want every single thing that is contained in the library; the CPacked library may include e.g. test binaries that we do not want to include in the installation.
  • I don't know enough about how scikit-build-core configures CMake on the back end to know if we can safely reuse its build directory for the C++ build, otherwise we could simply run CMake commands directly in there and specify install components to split up the build. I doubt this will "just work" out of the box, though, and I don't know if we will ever be able to expect this to be a supported mode of operation for scikit-build-core even if we could make it work.
  • One possible option would be for the C++ and C++ wheel builds to be done in the same CI job so that the C++ build happens in a build directory that is still available when doing the wheel build. That might allow us some more flexibility in doing the two step wheel build to pull everything more granularly from the raw C++ CMake build.

This may require some level of work on the scikit-build-core side to support.

@robertmaynard
Copy link

Associated issue: rapidsai/cuvs#449

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants