Installation fails during testing

[ozandernek]

Hi everyone,

I am trying to install TRIQS v3.0.2 using easybuild on a RH8.5 cluster. During the testing, 2/3 of the tests fails, almost all with the SEGFAULT error. Here is the full command I used:

eb -dr --buildpath=/tmp/9093_eb --optarch=march=native --umask=007 /usr/local/easybuild/software/EasyBuild/4.5.5/easybuild/easyconfigs/t/TRIQS/TRIQS-3.0.2-foss-2020a-Python-3.8.2.eb

I was advised to use the full path of the easyconfig by our cluster admin. I guess there can be lots of reasons for segmentation fault but I'll appreciate your help. Here is the last few lines from the log file:

Errors while running CTest
make: *** [Makefile: 100: test] Error 8
(at easybuild/software/EasyBuild/4.5.5/lib/python3.6/site-packages/easybuild/tools/run.py:618 in parse_cmd_output)
== 2022-06-23 23:01:03,664 build_log.py:265 INFO ... (took 1 min 9 secs)
== 2022-06-23 23:01:03,664 config.py:660 DEBUG software install path as specified by 'installpath' and 'subdir_software': /home/9093/.local/easybuild/software
== 2022-06-23 23:01:03,665 filetools.py:1983 INFO Removing lock /home/9093/.local/easybuild/software/.locks/home_9093.local_easybuild_software_TRIQS_3.0.2-foss-2020a-Python-3.8.2.l
ock...
== 2022-06-23 23:01:03,666 filetools.py:382 INFO Path /home/9093/.local/easybuild/software/.locks/home_9093.local_easybuild_software_TRIQS_3.0.2-foss-2020a-Python-3.8.2.lock succes
sfully removed.
== 2022-06-23 23:01:03,666 filetools.py:1987 INFO Lock removed: /home/9093/.local/easybuild/software/.locks/home_9093.local_easybuild_software_TRIQS_3.0.2-foss-2020a-Python-3.8.2.l
ock
== 2022-06-23 23:01:03,667 easyblock.py:4079 WARNING build failed (first 300 chars): cmd " make test " exited with exit code 2 and output:
Running tests...
/usr/local/easybuild/software/CMake/3.16.4-GCCcore-9.3.0/bin/ctest --force-new-ctest-process
Test project /tmp/9093_eb/TRIQS/3.0.2/foss-2020a-Python-3.8.2/easybuild_obj
Start 1: itertools
1/209 Test #1: itertools
== 2022-06-23 23:01:03,667 easyblock.py:318 INFO Closing log for application name TRIQS version 3.0.2

[the-hampel]

Hi @ozandernek,

I did not use the easybuild scripts yet, so maybe I cannot help solving this directly. I know that we recently used the easybuild scripts for triqs version 3.1.x successfully. So it should work in principle.

To better understand what goes wrong, I would like to see why one of the tests fail. I would assume that if that many fail it might be related to the same problem. So the error output from only one of the tests would be helpful. If it is not shown directly in the easybuild log you might find more info in the build directory of triqs (/tmp/9093_eb/TRIQS/3.0.2/foss-2020a-Python-3.8.2/easybuild_obj here?). You can look explicitly in the Testing/Temporary/ folder. There should be some text files containing the output of the test run.

Best,
Alex

[ozandernek]

Hi @the-hampel,

You did more than enough! It turns out some of the tests require OpenMP access with multiple processors. So I put the installation command on a job script and submit the script via slurm scheduler. I also had to purge all my loaded modules and reload EasyBuild module. Here is what it looks like, if anyone needs it in the future:

#!/bin/bash
#SBATCH -c 8
#SBATCH --mem=24G
#SBATCH -o TRIQS_build.log

module purge
module load EasyBuild

umask 002
eb -dr --umask=002 --buildpath=/SOME/LOCAL/PATH --optarch=GENERIC /FULL/PATH/FOR/TRIQS/TRIQS-3.0.2-foss-2020a-Python-3.8.2.eb

[Wentzell]

Dear @ozandernek @the-hampel,

There should be no openmp required to run the TRIQS tests, so I don't quite understand what went wrong in your build.

But do I understand correctly that your build is now successfully passing?

Regarding the 3.1 recipes, those are unfortunately not yet merged into the main branch EasyBuild.
It should be part of the next release however, c.f. easybuilders/easybuild-easyconfigs#15183

[ozandernek]

Hi @Wentzell,

Yes, my last attempt was successful. The OpenMP issue might be specific to our cluster, because of the way easybuild was setup. I reproduced the error just to give an example, and below is the note from the log file for one of the failed tests:

2/209 Testing: comm_split_np2
2/209 Test: comm_split_np2
Command: "/usr/local/easybuild/software/OpenMPI/4.0.3-GCC-9.3.0/bin/mpiexec" "-n" "4" "--oversubscribe" "/tmp/9093_eb/TRIQS/3.0.2/foss-2020a-Python-3.8.2/easybuild_obj/deps/mpi/test/c+
+//comm_split"
Directory: /tmp/9093_eb/TRIQS/3.0.2/foss-2020a-Python-3.8.2/easybuild_obj/deps/mpi/test/c++/
"comm_split_np2" start time: Jun 27 13:16 EDT
Output:

---

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpiexec noticed that process rank 2 with PID 0 on node hpc7 exited on signal 11 (Segmentation fault).

Test time = 3.12 sec ---------------------------------------------------------- Test Failed. "comm_split_np2" end time: Jun 29 5:42 EDT "comm_split_np2" time elapsed: 00:00:03 ----------------------------------------------------------

[the-hampel]

Ah this is about OpenMPI not OpenMP. As @Wentzell correctly pointed out, there is no OpenMP dependency atm. That the tests fail on a login node on a cluster is a known problem. We can maybe add a note about this in the installation instructions. This is because it is usually forbidden to spawn MPI on the login node without having a allocation. But to make this clear, this is not a problem with TRIQS, this is a general problem specific to clusters.

But very good that everything works now.

Best,
Alex