Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prefer wheel-provided libcuspatial.so in load_library(), use RTLD_LOCAL #1483

Merged
merged 5 commits into from
Nov 14, 2024

Conversation

jameslamb
Copy link
Member

@jameslamb jameslamb commented Nov 13, 2024

Description

Contributes to rapidsai/build-planning#118

Modifies libcuspatial.load_library() in the following ways:

  • prefer wheel-provided libcuspatial.so to system installation
  • expose environment variable RAPIDS_LIBCUSPATIAL_PREFER_SYSTEM_LIBRARY for switching that preference
  • load libcuspatial.so with RTLD_LOCAL, to prevent adding symbols to the global namespace (dlopen docs)

Notes for Reviewers

How I tested this

See "How I tested this" in rapidsai/cudf#17316

Also opened this PR originally pulling in built packages from rapidsai/cudf#17316

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@jameslamb jameslamb added 5 - DO NOT MERGE Hold off on merging; see PR for details improvement Improvement / enhancement to an existing function breaking Breaking change labels Nov 13, 2024
@github-actions github-actions bot added the Python Related to Python code label Nov 13, 2024
Copy link
Contributor

@vyasr vyasr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general logic looks right. Rewrote the explanatory comment to hopefully clarify what we're doing, and left one comment on resolution order.

python/libcuspatial/libcuspatial/load.py Outdated Show resolved Hide resolved
python/libcuspatial/libcuspatial/load.py Show resolved Hide resolved
@jameslamb jameslamb requested a review from vyasr November 14, 2024 00:33
@jameslamb jameslamb changed the title WIP: [DO NOT MERGE] prefer wheel-provided libcudf.so in load_library(), use RTLD_LOCAL prefer wheel-provided libcudf.so in load_library(), use RTLD_LOCAL Nov 14, 2024
@jameslamb jameslamb removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Nov 14, 2024
@jameslamb jameslamb marked this pull request as ready for review November 14, 2024 21:06
@jameslamb jameslamb requested a review from a team as a code owner November 14, 2024 21:06
@jameslamb jameslamb requested review from harrism and isVoid November 14, 2024 21:06
@jameslamb jameslamb changed the title prefer wheel-provided libcudf.so in load_library(), use RTLD_LOCAL prefer wheel-provided libcuspatial.so in load_library(), use RTLD_LOCAL Nov 14, 2024
@vyasr
Copy link
Contributor

vyasr commented Nov 14, 2024

/merge

@rapids-bot rapids-bot bot merged commit 20abaaa into rapidsai:branch-24.12 Nov 14, 2024
70 checks passed
@jameslamb jameslamb deleted the library-loading branch November 14, 2024 22:04
rapids-bot bot pushed a commit to rapidsai/kvikio that referenced this pull request Nov 14, 2024
Contributes to rapidsai/build-planning#118

Modifies `libkvikio.load_library()` in the following ways:

* prefer wheel-provided `libkvikio.so` to system installation
* expose environment variable `RAPIDS_LIBKVIKIO_PREFER_SYSTEM_LIBRARY` for switching that preference
* load `libkvikio.so` with `RTLD_LOCAL`, to prevent adding symbols to the global namespace ([dlopen docs](https://linux.die.net/man/3/dlopen))

## Notes for Reviewers

### How I tested this

Tested the general approach on rapidsai/cudf#17316 and rapidsai/cuspatial#1483.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #551
rapids-bot bot pushed a commit that referenced this pull request Nov 15, 2024
Contributes to rapidsai/build-planning#118

The pattern introduced in #1483 breaks editable installs in devcontainers. In that type of build, `libcuspatial.so` is built outside of the wheel but **not installed**, so it can't be found by `ld`. Extension modules in `cuspatial` are able to find it via RPATHs instead.

This proposes:

* try-catching the entire library-loading attempt, to silently do nothing in cases like that
* ~adding an import of the `cuspatial` Python library in the `devcontainers` CI job, as a smoke test to catch issues like this in the future~ *(edit: removed those, [`devcontainer` builds run on CPU nodes](https://github.com/rapidsai/shared-workflows/blob/4e84062f333ce5649bc65029d3979569e2d0a045/.github/workflows/build-in-devcontainer.yaml#L19))*

## Notes for Reviewers

### How I tested this

Tested this approach on rapidsai/kvikio#553

#

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #1484
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Breaking change improvement Improvement / enhancement to an existing function Python Related to Python code
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

3 participants