Build error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. for test_la.py #73
Comments
|
I search around a little bit, and from intel website, it might have to do with an error in mkl, specifically with lmkl_blacs_intelmpi_lp64. I do see a dynamic version of lmkl_blacs_intelmpi_lp64 in my path though, the comment from running configure indicates that; Checking for dynamic BLAS library... dynamic BLAS library found, with underscores. The error seems to originate from test_la, which I presume is due to testing SVD? |
|
@AdrianTanTeckKeng thanks for reporting this and sorry for the slow response, had missed the issue creation. I've been able to reproduce problems with this test and MKL recently and will try to sort things out. There has been a few problems associated with the usage of MKL ScaLAPACK, especially with complex kernels that seem tied to optimization flags and details of which MKL routine is being invoked. If this test is failing, much of everything might still work. Will try to investigate and update in the next few days. |
|
I see, does it mean that if I build my own scalapack library using the configure option, one technically will avoid this problem? I did try that but ran into errors doing so, will try it again later. |
|
Possibly, but not certainly. And indeed getting any combination of dynamic MKL/ScaLAPACK libraries to work nicely in a supercomputing setup is non-trivial, even more so when going through python for setup and linking. I started playing around with this earlier this week after seeing your email, but was also not yet able to resolve the problem. I think I may have successfully made it go away on Stampede2 in the past though, so will keep trying and get back to you. |
…o we redistribute to nearest square processor grid. Some bugs associated with communication from/to subworlds are made to get this morning. Its possible these fixes would address issue #73 but the error message appearing there seems different.
Hi all, I am wondering if there is support provided for compiling on SDSC comet(I used similar compiling strategy listed on the github wiki for Stampede2). I loaded the modules below and managed to run all 56 tests successfully for make test. The problem comes in test_la.py during make python_test. I saw a comment here that indicates there might be a problem with eigh for python interface? The comment is unclear what is failing though. Anyway, I will post my error log here. I also listed details of mpicxx and python below:
-------------------------------------Build details-------------------------------------------
modules loaded on SDSC Comet
Currently Loaded Modulefiles:
mpicxx version
mpiicpc for the Intel(R) MPI Library 5.1.3 for Linux*
Copyright(C) 2003-2015, Intel Corporation. All rights reserved.
icpc version 16.0.3 (gcc version 4.9.2 compatibility)
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Build command
./configure --no-dynamic LIB_PATH="-L$MKLROOT/lib/intel64/" LIBS="-lmkl_scalapack_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -lpthread -lm"
----------------------Error log during make python portion----------------------------------
cd src_python;
ln -sf /home/atanteck/ctf/ctf/setup.py setup.py;
mkdir -p /home/atanteck/ctf/ctf/lib_python/ctf && cp ctf/init.py /home/atanteck/ctf/ctf/lib_python/ctf/;
LDFLAGS="-L/home/atanteck/ctf/ctf/lib_shared" python setup.py build_ext --force -b /home/atanteck/ctf/ctf/lib_python/ -t /home/atanteck/ctf/ctf/lib_python/;
rm setup.py;
cd ..;
running build_ext
building 'ctf.core' extension
mpicxx -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ictf -I/home/atanteck/anaconda3/lib/python3.7/site-packages/numpy/core/include -I/home/atanteck/ctf/ctf/include -I. -I/home/atanteck/anaconda3/include/python3.7m -c ctf/core.cpp -o /home/atanteck/ctf/ctf/lib_python/ctf/core.o -qopenmp -O3 -ipo -std=c++0x
In file included from /home/atanteck/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h(1824),
from /home/atanteck/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h(12),
from /home/atanteck/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h(4),
from ctf/core.cpp(630):
/home/atanteck/anaconda3/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h(17): warning #1224: #warning directive: "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
#warning "Using deprecated NumPy API, disable it with "
^
mpicxx -pthread -shared -B /home/atanteck/anaconda3/compiler_compat -L/home/atanteck/anaconda3/lib -Wl,-rpath=/home/atanteck/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ -L/home/atanteck/ctf/ctf/lib_shared /home/atanteck/ctf/ctf/lib_python/ctf/core.o -lctf -lmkl_scalapack_lp64 -l-Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -l-Wl,--end-group -lpthread -lm -o /home/atanteck/ctf/ctf/lib_python/ctf/core.cpython-37m-x86_64-linux-gnu.so -L/home/atanteck/ctf/ctf/lib_shared -qopenmp -O3 -ipo -std=c++0x -L/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/ -Wl,-rpath=/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/
ipo: warning #11012: unable to find -l-Wl,--start-group
ipo: warning #11012: unable to find -l-Wl,--end-group
": internal error: 010101_(5000 + 54)
icpc: error #10014: problem during multi-file optimization compilation (code 4)
error: command 'mpicxx' failed with exit status 4
LD_LIBRARY_PATH="/opt/gnu/gcc/lib64:/opt/gnu/gmp/lib:/opt/gnu/mpfr/lib:/opt/gnu/mpc/lib:/opt/intel/2016.3.210/compilers_and_libraries/linux/mpi/intel64/lib:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mpirt/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/../compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4:/opt/sdsc/lib:/home/atanteck/ctf/ctf/lib_shared:/home/atanteck/ctf/ctf/lib_python:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/" PYTHONPATH=":/home/atanteck/ctf/ctf/lib_python" python ./test/python/test_base.py
Tests for basic numpy ndarray functionality
./test/python/test_base.py:89: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
self.assertTrue(ctf.all(a0==a1))
Ran 21 tests in 0.121s
OK
LD_LIBRARY_PATH="/opt/gnu/gcc/lib64:/opt/gnu/gmp/lib:/opt/gnu/mpfr/lib:/opt/gnu/mpc/lib:/opt/intel/2016.3.210/compilers_and_libraries/linux/mpi/intel64/lib:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mpirt/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/../compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4:/opt/sdsc/lib:/home/atanteck/ctf/ctf/lib_shared:/home/atanteck/ctf/ctf/lib_python:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/" PYTHONPATH=":/home/atanteck/ctf/ctf/lib_python" python ./test/python/test_fancyindex.py
Tests for fancy index
Ran 8 tests in 0.063s
OK
LD_LIBRARY_PATH="/opt/gnu/gcc/lib64:/opt/gnu/gmp/lib:/opt/gnu/mpfr/lib:/opt/gnu/mpc/lib:/opt/intel/2016.3.210/compilers_and_libraries/linux/mpi/intel64/lib:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mpirt/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/../compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4:/opt/sdsc/lib:/home/atanteck/ctf/ctf/lib_shared:/home/atanteck/ctf/ctf/lib_python:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/" PYTHONPATH=":/home/atanteck/ctf/ctf/lib_python" python ./test/python/test_einsum.py
Tests for einsum
F
FAIL: test_tree_ctr (main.KnowValues)
Traceback (most recent call last):
File "./test/python/test_einsum.py", line 175, in test_tree_ctr
self.assertTrue(numpy.abs(scl-scl2)<1.e-4)
AssertionError: False is not true
Ran 8 tests in 0.313s
FAILED (failures=1)
LD_LIBRARY_PATH="/opt/gnu/gcc/lib64:/opt/gnu/gmp/lib:/opt/gnu/mpfr/lib:/opt/gnu/mpc/lib:/opt/intel/2016.3.210/compilers_and_libraries/linux/mpi/intel64/lib:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mpirt/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/../compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4:/opt/sdsc/lib:/home/atanteck/ctf/ctf/lib_shared:/home/atanteck/ctf/ctf/lib_python:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/" PYTHONPATH=":/home/atanteck/ctf/ctf/lib_python" python ./test/python/test_ufunc.py
Tests for univeral functions
............./test/python/test_ufunc.py:31: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
self.assertTrue(ctf.all(a0==a1))
Ran 16 tests in 0.077s
OK
LD_LIBRARY_PATH="/opt/gnu/gcc/lib64:/opt/gnu/gmp/lib:/opt/gnu/mpfr/lib:/opt/gnu/mpc/lib:/opt/intel/2016.3.210/compilers_and_libraries/linux/mpi/intel64/lib:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mpirt/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/../compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4:/opt/sdsc/lib:/home/atanteck/ctf/ctf/lib_shared:/home/atanteck/ctf/ctf/lib_python:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/" PYTHONPATH=":/home/atanteck/ctf/ctf/lib_python" python ./test/python/test_dot.py;
Tests for dot
Ran 4 tests in 0.045s
OK
LD_LIBRARY_PATH="/opt/gnu/gcc/lib64:/opt/gnu/gmp/lib:/opt/gnu/mpfr/lib:/opt/gnu/mpc/lib:/opt/intel/2016.3.210/compilers_and_libraries/linux/mpi/intel64/lib:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mpirt/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/../compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4:/opt/sdsc/lib:/home/atanteck/ctf/ctf/lib_shared:/home/atanteck/ctf/ctf/lib_python:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/" PYTHONPATH=":/home/atanteck/ctf/ctf/lib_python" python ./test/python/test_sparse.py;
Tests for sparse functionality
Ran 5 tests in 0.048s
OK
LD_LIBRARY_PATH="/opt/gnu/gcc/lib64:/opt/gnu/gmp/lib:/opt/gnu/mpfr/lib:/opt/gnu/mpc/lib:/opt/intel/2016.3.210/compilers_and_libraries/linux/mpi/intel64/lib:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mpirt/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/../compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4:/opt/sdsc/lib:/home/atanteck/ctf/ctf/lib_shared:/home/atanteck/ctf/ctf/lib_python:/share/apps/compute/intel/intelmpi/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/" PYTHONPATH=":/home/atanteck/ctf/ctf/lib_python" python ./test/python/test_la.py;
Tests for linear algebra functionality
Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.
make: *** [python_la_test] Error 2
The text was updated successfully, but these errors were encountered: