[BUG]: Python thread finalizer can cause thread-local storage to be cleared prematurely in some situations #362
Assignees
Labels
bug
Something isn't working
Comments
3 tasks
dagardner-nv
added a commit
to dagardner-nv/MRC
that referenced
this issue
Aug 17, 2023
…age collector prior to finalizing the thread
3 tasks
dagardner-nv
added a commit
to dagardner-nv/MRC
that referenced
this issue
Aug 17, 2023
…hon garbage collector prior to finalizing the thread" This reverts commit 50abe96.
This was referenced Aug 17, 2023
rapids-bot bot
pushed a commit
to nv-morpheus/utilities
that referenced
this issue
Aug 17, 2023
* Needed for nv-morpheus/MRC#362 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #52
dagardner-nv
added a commit
to dagardner-nv/MRC
that referenced
this issue
Aug 18, 2023
dagardner-nv
added a commit
to dagardner-nv/MRC
that referenced
this issue
Aug 18, 2023
…v-morpheus#362" This reverts commit 11c93a6.
3 tasks
rapids-bot bot
pushed a commit
that referenced
this issue
Sep 25, 2023
* PR replicates issue #362, and will trigger a pybind11 internal error using an un-patched version of pybind11 * Only run IWYU on files changed in PR Note: * This bug requires the code in question to be run in a thread created by C++. fixes #362 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #365
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
23.11
Which installation method(s) does this occur on?
Docker, Conda, Source
Describe the bug.
When the thread finalizer (https://github.com/nv-morpheus/MRC/blob/branch-23.11/python/mrc/_pymrc/src/executor.cpp#L125) is called it decrements the GIL which releases all objects in thread local storage.
The issue is that if once of the objects being released also declares a finalizer (https://github.com/cupy/cupy/blob/fd54d95af0110e6bdf840da7a833357e72fbb43b/cupy/fft/_cache.pyx#L152) which in turn makes a call to
gc.collect(https://github.com/cupy/cupy/blob/fd54d95af0110e6bdf840da7a833357e72fbb43b/cupy/fft/_cache.pyx#L128) then garbage collection runs while the thread state is in the middle of being destroyed.Minimum reproducible example
No response
Relevant log output
No response
Full env printout
No response
Other/Misc.
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: