Setting environment variables for context_python

[JamesWrigley]

@daviddoji and I ran into a situation where using a custom environment with a package that has binary dependencies (numba) threw confusing load errors:

The problem is that the amore modules set the LD_LIBRARY_PATH variables, which caused Python to find an incompatible libstdc++ from the amore/mid module instead of the one that the offending shared object (libLLVM-14.so) was actually linked against in our current environment:

wrigleyj@max-exfl462 /gpfs/exfel/sw/software/mambaforge/22.11/envs/202402/lib % ldd libLLVM-14.so
        linux-vdso.so.1 (0x00007fff237d3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x000014fbe8280000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x000014fbe827b000)
        libz.so.1 => /gpfs/exfel/sw/software/mambaforge/22.11/envs/202402/lib/././libz.so.1 (0x000014fbe8260000)
        libstdc++.so.6 => /gpfs/exfel/sw/software/mambaforge/22.11/envs/202402/lib/././libstdc++.so.6 (0x000014fbe807d000)
        libm.so.6 => /lib64/libm.so.6 (0x000014fbe7fa2000)
        libgcc_s.so.1 => /gpfs/exfel/sw/software/mambaforge/22.11/envs/202402/lib/././libgcc_s.so.1 (0x000014fbe7f81000)
        libc.so.6 => /lib64/libc.so.6 (0x000014fbe7c00000)
        /lib64/ld-linux-x86-64.so.2 (0x000014fbef307000)
        librt.so.1 => /lib64/librt.so.1 (0x000014fbe7f7c000)

Presumably the child context_python process is inheriting the environment variables from the parent, so we hacked around it with a shim:

#! /usr/bin/bash

export LD_LIBRARY_PATH=/gpfs/exfel/sw/software/mambaforge/22.11/envs/202402/lib
exec /gpfs/exfel/u/usr/SXP/202431/p900460/Software/damnit_env/bin/python "$@"

But it would be nice if there was a way to specify environment variables for context_python, or a command to activate the environment properly.

[takluyver]

Curiously, import pandas, numba hits that error, but import numba, pandas doesn't. So loading pandas is setting up some global state which is part of the problem. 😕