Merge fails in Pre-processing when picking particles

[JuhaHuiskonen]

I completed first round of preprocessing without errors. I have now picked particles based on size (radius) and running the pre-processing block again. The individual jobs complete without errors. However, the merge step fails with the following error

1	2024-02-03 12:05:10 [INFO] /opt/pyp/bin/run/pyp:3273 | Job 3154908 (v0.6.2) launching on c6231.mahti.csc.fi using 2 task(s) and 200 GB of RAM
2	2024-02-03 12:05:40 [WARNING] pyp:677 | Update micrograph/tilt-series order according to number of particles
3	2024-02-03 12:05:48 [INFO] pyp/refine/eman/eman.py:63 | export PYTHONPATH=/opt/eman2/pkgs; cd eman; e2proclst.py ../sva/lamella8_ts_008_*.rec --create lamella8_ts_008_subvols.lst
4	Traceback (most recent call last):
5	  File "/opt/pyp/bin/run/pyp", line 3518, in <module>
6	    tomo_merge(parameters)
7	  File "/opt/pyp/bin/run/pyp", line 861, in tomo_merge
8	    eman.eman_3davg(parameters)
9	  File "/opt/pyp/src/pyp/refine/eman/eman.py", line 65, in eman_3davg
10	    subprocess.check_output(
11	  File "/usr/local/envs/pyp/lib/python3.8/subprocess.py", line 415, in check_output
12	    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
13	  File "/usr/local/envs/pyp/lib/python3.8/subprocess.py", line 516, in run
14	    raise CalledProcessError(retcode, process.args,
15	subprocess.CalledProcessError: Command 'export PYTHONPATH=/opt/eman2/pkgs; cd eman; e2proclst.py ../sva/lamella8_ts_008_*.rec --create lamella8_ts_008_subvols.lst' returned non-zero exit status 127.
16	2024-02-03 12:05:48 [ERROR] /opt/pyp/bin/run/pyp:3525 | PYP (tomomerge) failed

I have checked and the input files ../sva/lamella8_ts_008_*.rec are present. Suggestions how to fix this error?

Edit: lamella8_ts_008 is the one with the largest number of picked particles.

Edit2: Originally the format was eman. I changed the extraction format 3davg and it worked. Changing back to eman for testing purposes repeats the error, so this was not merely fixed by re-extracting the particles, but seems to be specific to eman with my data

[abartesaghi]

This is the very last step of the tomomerge job where it tries to prepare the inputs needed to run sub-tomogram averaging in eman2 using the e2proclst.py script. However, since eman2 is not part of nextpyp, this step fails simply because the script is not there. You can safely ignore the error and execute the script manually outside of nextpyp (assuming you have a working installation of eman2 in your system). You will need to cd to the project folder and run:

cd eman; e2proclst.py ../sva/lamella8_ts_008_*.rec --create lamella8_ts_008_subvols.lst

Thanks for reporting this.

[JuhaHuiskonen]

What is the different between eman and avg3 in this step? Can either of them be used? When I use avg3 I get this error in refinement FileNotFoundError: [Errno 2] No such file or directory: 'stacks.txt' and I wonder if this is related.

[abartesaghi]

The only difference between eman and 3davg is that the contrast of the sub-tomograms is inverted (protein=white for eman). The FileNotFoundError error seems to be unrelated to sub-tomogram extraction. It will help to know more about what you are trying to do.

[JuhaHuiskonen]

Thanks. I am trying to start my first Particle averaging run with my data. This is the full error:

1	2024-02-03 18:29:40 [INFO] /opt/pyp/bin/run/pyp:3273 | Job 3156009_1 (v0.6.2) launching on c1246.mahti.csc.fi using 128 task(s) and 200 GB of RAM
2	2024-02-03 18:29:40 [INFO] /opt/pyp/bin/run/pyp:3200 | Filesystem                                  Size  Used Avail Use% Mounted on
3	2024-02-03 18:29:40 [INFO] /opt/pyp/bin/run/pyp:3200 | 10.141.0.14@o2ib:10.141.0.13@o2ib:/scratch   40T  881G   40T   3% /scratch/project_2009485/nextPYP/tmp
4	2024-02-03 18:29:40 [INFO] pyp/system/mpi.py:180 | First function is: load_files
5	2024-02-03 18:29:41 [INFO] pyp/system/mpi.py:180 | First function is: load_files
6	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009.webp
7	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009.rec
8	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009_ali.webp
9	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009_rec.webp
10	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009.mrc
11	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009.pkl
12	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009_sides.webp
13	2024-02-03 18:29:41 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_009_raw.webp
14	2024-02-03 18:30:17 [INFO] pyp:1795 | Executing coordinate extraction
15	2024-02-03 18:30:17 [INFO] pyp:57 | Reading and converting coordinates took: 00h 00m 00s
16	2024-02-03 18:30:17 [INFO] /opt/pyp/bin/run/pyp:57 | Total time elapsed (csp_swarm): 00h 00m 37s
17	Traceback (most recent call last):
18	  File "/opt/pyp/bin/run/pyp", line 3914, in <module>
19	    csp_swarm(args.file, parameters, int(args.iter), args.skip, args.debug)
20	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
21	    return func(*args, **kwargs)
22	  File "/opt/pyp/bin/run/pyp", line 2376, in csp_swarm
23	    [allboxes, allparxs] = csp_extract_coordinates(
24	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
25	    return func(*args, **kwargs)
26	  File "/opt/pyp/src/pyp/inout/metadata/core.py", line 1869, in csp_extract_coordinates
27	    [allboxes, allparxs] = tomo_extract_coordinates(
28	  File "/opt/pyp/src/pyp/inout/metadata/core.py", line 2007, in tomo_extract_coordinates
29	    refinement = project_params.resolve_path(parameters["refine_parfile_tomo"])
30	KeyError: 'refine_parfile_tomo'
31	2024-02-03 18:30:17 [ERROR] /opt/pyp/bin/run/pyp:3927 | PYP (cspswarm) failed
32	2024-02-03 18:30:21 [INFO] /opt/pyp/bin/run/pyp:3273 | Job 3156009_1 (v0.6.2) launching on c1246.mahti.csc.fi using 128 task(s) and 200 GB of RAM
33	2024-02-03 18:30:23 [INFO] /opt/pyp/bin/run/pyp:3200 | Filesystem                                  Size  Used Avail Use% Mounted on
34	2024-02-03 18:30:23 [INFO] /opt/pyp/bin/run/pyp:3200 | 10.141.0.14@o2ib:10.141.0.13@o2ib:/scratch   40T  1.1T   39T   3% /scratch/project_2009485/nextPYP/tmp
35	2024-02-03 18:30:23 [INFO] pyp/system/mpi.py:180 | First function is: load_files
36	2024-02-03 18:30:24 [INFO] pyp/system/mpi.py:180 | First function is: load_files
37	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002.pkl
38	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002_raw.webp
39	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002.rec
40	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002_sides.webp
41	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002.webp
42	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002.mrc
43	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002_rec.webp
44	2024-02-03 18:30:25 [INFO] pyp/inout/utils/core.py:17 | Retrieving GS8_TS27_ts_002_ali.webp
45	2024-02-03 18:30:48 [INFO] pyp:1795 | Executing coordinate extraction
46	2024-02-03 18:30:48 [INFO] pyp:57 | Reading and converting coordinates took: 00h 00m 00s
47	2024-02-03 18:30:48 [INFO] /opt/pyp/bin/run/pyp:57 | Total time elapsed (csp_swarm): 00h 00m 24s
48	Traceback (most recent call last):
49	  File "/opt/pyp/bin/run/pyp", line 3914, in <module>
50	    csp_swarm(args.file, parameters, int(args.iter), args.skip, args.debug)
51	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
52	    return func(*args, **kwargs)
53	  File "/opt/pyp/bin/run/pyp", line 2376, in csp_swarm
54	    [allboxes, allparxs] = csp_extract_coordinates(
55	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
56	    return func(*args, **kwargs)
57	  File "/opt/pyp/src/pyp/inout/metadata/core.py", line 1869, in csp_extract_coordinates
58	    [allboxes, allparxs] = tomo_extract_coordinates(
59	  File "/opt/pyp/src/pyp/inout/metadata/core.py", line 2007, in tomo_extract_coordinates
60	    refinement = project_params.resolve_path(parameters["refine_parfile_tomo"])
61	KeyError: 'refine_parfile_tomo'
62	2024-02-03 18:30:48 [ERROR] /opt/pyp/bin/run/pyp:3927 | PYP (cspswarm) failed
63	2024-02-03 18:30:51 [INFO] /opt/pyp/bin/run/pyp:3273 | Job 3156009_1 (v0.6.2) launching on c1246.mahti.csc.fi using 128 task(s) and 200 GB of RAM
64	2024-02-03 18:30:51 [INFO] pyp/refine/csp/particle_cspt.py:57 | Total time elapsed (csp_local_merge): 00h 00m 00s
65	Traceback (most recent call last):
66	  File "/opt/pyp/bin/run/pyp", line 3992, in <module>
67	    particle_cspt.merge_movie_files_in_job_arr(
68	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
69	    return func(*args, **kwargs)
70	  File "/opt/pyp/src/pyp/refine/csp/particle_cspt.py", line 380, in merge_movie_files_in_job_arr
71	    with open(movie_file) as f:
72	FileNotFoundError: [Errno 2] No such file or directory: 'stacks.txt'
73	2024-02-03 18:30:51 [ERROR] /opt/pyp/bin/run/pyp:4012 | PYP (csp_local_merge) failed
74	2024-02-03 18:30:52 [INFO] /opt/pyp/bin/run/pyp:4018 | Deleted temporary files from /scratch/project_2009485/nextPYP/tmp/huiskone/3156009_1

These are the parameters of the refinement section:

[abartesaghi]

Step 6 in the tomography tutorial seems to be out of date. You have to provide the Alignments from sub-volume averaging parameter (note that this is an advanced parameter so you need to show the advanced options) by selecting the location of the txt file tomo-preprocessing-*_original_volumes.txt. Just to clarify, you don't need to extract sub-tomograms for running 3D refinement.

[JuhaHuiskonen]

Thanks, this helped and the refinement seems to be running now.