IMOD segfaults in patch tracking

[JuhaHuiskonen]

Hi all, I am experiencing IMOD segfaults in patch tracking. Not strictly a nextPYP question, but if anyone has seen this and has a solution, please let me know. These occur quite randomly. For instance, have of the tilt series might segfault in the first pass, but then complete fine in the second pass.

83	2024-02-08 19:28:14 [INFO] pyp/align/core.py:5312 | Align tilt-series using patch tracking (IMOD)
84	2024-02-08 19:28:58 [ERROR] pyp/system/local_run.py:72 | forrtl: severe (174): SIGSEGV, segmentation fault occurred
85	Image              PC                Routine            Line        Source             
86	libifcoremt.so.5   00007FFFBD9B5F0C  for__signal_handl     Unknown  Unknown
87	libpthread-2.17.s  00007FFFBB1EA630  Unknown               Unknown  Unknown
88	libiomp5.so        00007FFFBB76B831  Unknown               Unknown  Unknown
89	libiomp5.so        00007FFFBB76CD72  Unknown               Unknown  Unknown
90	libifcoremt.so.5   00007FFFBD9F8C76  for_allocate          Unknown  Unknown
91	tiltalign          000000000040521B  Unknown               Unknown  Unknown
92	tiltalign          0000000000402D3E  Unknown               Unknown  Unknown
93	libc-2.17.so       00007FFFBAE2F555  __libc_start_main     Unknown  Unknown
94	tiltalign          0000000000402C49  Unknown               Unknown  Unknown
95	
96	2024-02-08 19:28:58 [INFO] pyp/align/core.py:57 | Alignment took: 00h 01m 00s

I have tested number of patches 6-10 and binning 4-8. I am using 8 threads per task and 200 GB memory. Any suggestions how to prevent segfaults?

[JuhaHuiskonen]

Looks like it is a memory issue. Running with 1-4 threads seems to resolve the issue.

[JuhaHuiskonen]

Sadly this was not the solution. About half of the tilt series still give a segmentation fault. It is strange that these worked earlier with the default IMOD settings, so something is different now, but still could not figure out what. I am reserving 200 GB memory and command sacct shows that the maximum memory usage has been less than 50GB in all of the jobs.

[abartesaghi]

I don't think tiltalign in particular uses a lot of space, but one thing to check is whether you have enough scratch space on all the nodes. This information should be listed at the beginning of each job. If I remember correctly, your setup used gres to request local scratch, so this may not actually be an issue in your case, but worth checking anyways.

[JuhaHuiskonen]

I installed nextPYP on a different HPC environment. The nextPYP configuration is identical, but the configuration of the compute nodes is a bit different. Seems to run there fine. I wonder is it possible if one of the unblur runs does not finish (log is printed only for one movie) and then this would cause IMOD to crash as one of its input files is missing?

[abartesaghi]

You can check the logs for the other movies by going to the Logs tab, selecting the index number of the array job, and clicking on Load. If an input to IMOD is missing, you will usually get a different error. If unblur is indeed failing, the logs should tell you that pretty explicitely.

[abartesaghi]

We were able to reproduce this problem ,and at least of us, it only happens when running jobs on the gpu. We will investigate further but wanted to give an update here.

[JuhaHuiskonen]

I confirm that the jobs complete with the same parameters on a different (but highly similar) HPC environment. These tests were all pure CPU jobs.

[JuhaHuiskonen]

Is there a a way to copy the results from IMOD patch tracking done on a different machine to my project (the main machine has the resources for particle refinement, but it cannot complete IMOD patch tracking because of the segmentation fault)? I have tried copying the project over and importing it (tomography from PYP) but this didn't work for some reason. I would like to continue the current project (where patch tracking failed) simply by copying the files over to the right directories.

[yezhou867]

If you copy ".pkl" in the pkl folder and all the ".mrc" files from the mrc folder to the same places in your new project. You should be able to replicate the same results.

[yezhou867]

The ".mrc" files probably are not necessary. Just need the "pkl" files.

[JuhaHuiskonen]

Here's what I did and it worked:

1. Start the pre-processing job (which I know will lead to segfault in half of the tilt series)
2. Stop the job as soon as the split steps starts
3. Copy mrc rec webp and pkl files for all tilt series from the other HPC system (where the segfault error does not occur)
4. Rerun the job. Segfault occurs as expected during the first pass. But during the second pass it finds the copied files, and the job completes successfully.

[JuhaHuiskonen]

Hmm, the pre-processing/frealign folder was missing the original_volumes.txt file needed for refinement. I copied it over.

[abartesaghi]

Our best guess is that IMOD's tiltxcorr is failing to run under some conditions (this could be related on the specific node configuration somehow). The errors indicate that the solution to patch tracking is degenerate and this is causing problems downstream. The workaround we found was to tweak the number and the size of patches used by tiltxcorr (which can be done starting with version 0.6.3 of nextPYP).