Pip and Conda mixing

Dependency conflict

Suppose you have a Conda environment 'test' with Python and NumPy already installed (through conda install), and then have installed triumvirate using Pip. When validating the installation with

python -c 'from triumvirate import validate_installation as vi; vi()'

you might encounter dependency library conflicts such as this one with OpenMP:

OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/

You then try the suggested hack:

export KMP_DUPLICATE_LIB_OK=TRUE

but the validation either hangs, or results in some other error.

What's going on?

Her's some reading for you:

• Conflicting binary extensions in different packages
• BLAS, LAPACK and OpenMP

What's the solution?

Ideally, always follow this:

Tip

Do not mix Pip and Conda installation if it is avoidable. Stick to Conda as your package manager.

If you insist on a hack, use the OpenMP library provided in the Conda environment, and make a symbolic link from the path loaded by the Pip-installed package to the Conda one.

Demo

On a macOS machine, suppose I have the OpenMP shared library ${CONDA_PREFIX}/lib/libomp.dylib from my Conda environment, but my Pip-installed triumvirate loads ${CONDA_PREFIX}/lib/python3.10/site-packages/triumvirate/.dylibs/libomp.dylib instead, which comes bundled in the distributed wheel.

First, back up the OpenMP shared library bundled by the Pip-installation wheel with a different filename:

pip_bundled_libomp=${CONDA_PREFIX}/lib/python3.10/site-packages/triumvirate/.dylibs/libomp.dylib
mv ${pip_bundled_libomp} ${pip_bundled_libomp}.bak

Then, make a symbolic link from the old path to the Conda-installed OpenMP library:

conda_env_libomp=${CONDA_PREFIX}/lib/libomp.dylib
ln -s ${conda_env_libomp} ${pip_bundled_libomp}

Voila, your installation validation should work now, but don't forget the lesson above!

To restore the change made, simply:

rm ${pip_bundled_libomp}
mv ${pip_bundled_libomp}.bak ${pip_bundled_libomp}

A similar solution should work on Linux machines too.