Reference-based refinement failed

[kexinliu0001]

Hello, I am running the sample data (EMPIAR-10164) using standalone mode according to the tutorial. The previous steps worked fine, but the Reference-based refinement failed. I'm not sure what the problem is and how to deal with it. and I cannot find /opt/pyp/bin/run/pyp where the error seemed to be.

My setting is the same as shown in tutorial. the stream and logs is:

Moreover, I found that the requests for input parameter file in refinement tab is different from that in tutorial :

it asked for (*.par.bz2), and I found one in the folder where I downloaded the data for the tutorial

however, in tutorial the input parameter file should be (.txt)

Does the error above result from this difference?
if so, how can I select the text file from "pre-processing"?

[tofushan]

Sorry, the description is confusing. It is asking you to provide *.txt for the reference-based refinement instead of *.par.bz2 like other refinement blocks. We're making the distinction for the next release.

To select the txt file from upstream pre-processing block, you can first clear the selection by clicking the trash can icon, and then click the magnifier. It should direct you to the directory where the txt file is stored. If you still cannot find it, you can get the directory of the pre-processing block by selecting "Show filesystem location" of the block, and it should under {path_pre-processing}/frealign folder.

[kexinliu0001]

thank you for your reply!

I've resubmitted the job, and it is shown to be running.

however, 10 hours passed and the stream and log did not update. is it normal? it seemed to be stuck.

as I found there are 2 text files in the preprocessing folder:

is it ok to use just the second one when submitting the job?

[tofushan]

Regarding the file, you can always use tomo-preprocessing-xxxxx_volumes.txt, which is essentially a symbolic link pointing to the file with timestamps (the first one in your screenshot). So in your case, those two files are the same thing. If you re-run the pre-processing, a new txt file will be produced, and the link will also be updated to point to the most up-to-date txt file.

According to the log, it looks like refinement is running correctly. Unfortunately, as reference-based refinement is computationally expensive, the total running time is extremely long if using standalone mode. You may find similar discussion #23 helpful. We strongly suggest using SLURM in nextPYP for distributed processing if your computing environment allows you to do so.

[xingzhe1129]

Hi, I ran particle refienment for EMPIAR-10164 data set with 5 nodes and 4 array jobs ran successfully but always the first array job failed at csp_local_merge step. I repeated it above 5 times but the result was always the same. Do you meet this before? Do you know how to fix it? Thanks.

[yezhou867]

Was the first job always allocated to the same node or a random node? What's your slurm settings? The first job is always the one with the most particles which may need more memory than others. The error shows the output of the local merge job is missing. Can't tell too much whether it's because of memory or not.

[xingzhe1129]

Thank you so much for your quick response. The first job was not always allocated to the same node. The first job was always running the same TS_01 tomogram. The errors of the first job was not the same each time.

[yezhou867]

Do you have any information about the scratch space on each node? Looks like for the first jobs, the space could run out at a certain step.

[xingzhe1129]

Now our nextPYP temporary data was stored in /temp_work/cryosparc-wu , which is 5T and most of the steps before merging are running well but there is always a warning, which said "WARNING: While bind mounting '/temp_work/cryosparc-wu:/temp_work/cryosparc-wu': destination is already in the mount point list". Finally, the job failed, I am not sure whether the warning is the reason of job failure. Could you please take a look at the attachments and figure out the failure reason? Thank you so much.

[xingzhe1129]

The bundles size was 1 and I don’t know why it can’t run parallel. Before we set this new temporary file, it can run parallel

[yezhou867]

Probably need to set the slurm configuration in the right way by the root user so that each job will write in the scratch space with a different id name.
Temporarily, can try the bundle size of 5, so everything runs in a single job properly.

[xingzhe1129]

I tried with bundle size of 5 but it failed with the same reason. The error said that AssertionError: Number of intermediate reconstructions does not match (half1: 0 is NOT half2: 0), which is very similar to before with shortage of storage for temporary files. I can run Reference-based refinement and Fully constrained refinement successfully and I don't know why region-based local refinement failed. The attachment is the full config. Could you please take a look and help us figure it out? Thank you so much.

[yezhou867]

The configure file looks fine.
To understand the situation more clearly. Can you share more information about the following questions?

1. Did previous jobs also run with this new scratch space?
2. What is the output of the filter job before this region-based refinement?
3. If you could also provide a log from the successful fully constrained refinement job would be easier for us to compare. Since there should not be too much difference between those 2 jobs.

[xingzhe1129]

I tried to skip the filter job and ran region-based refinement successfully and will try to take a close look at the filter job. I like nextPYP and it will be perfect if we can ran all refinements with both GPUs and CPUs. Thanks.

[JuhaHuiskonen]

Hi, I am having the same error.

17	Traceback (most recent call last):
18	  File "/opt/pyp/bin/run/pyp", line 3914, in <module>
19	    csp_swarm(args.file, parameters, int(args.iter), args.skip, args.debug)
20	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
21	    return func(*args, **kwargs)
22	  File "/opt/pyp/bin/run/pyp", line 2376, in csp_swarm
23	    [allboxes, allparxs] = csp_extract_coordinates(
24	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
25	    return func(*args, **kwargs)
26	  File "/opt/pyp/src/pyp/inout/metadata/core.py", line 1869, in csp_extract_coordinates
27	    [allboxes, allparxs] = tomo_extract_coordinates(
28	  File "/opt/pyp/src/pyp/inout/metadata/core.py", line 2007, in tomo_extract_coordinates
29	    refinement = project_params.resolve_path(parameters["refine_parfile_tomo"])
30	KeyError: 'refine_parfile_tomo'
31	2024-02-08 12:09:38 [ERROR] /opt/pyp/bin/run/pyp:3927 | PYP (cspswarm) failed
32	2024-02-08 12:09:42 [INFO] /opt/pyp/bin/run/pyp:3273 | Job 3185360_1 (v0.6.2) launching on c1118.mahti.csc.fi using 128 task(s) and 200 GB of RAM
33	2024-02-08 12:09:42 [INFO] pyp/refine/csp/particle_cspt.py:57 | Total time elapsed (csp_local_merge): 00h 00m 00s
34	Traceback (most recent call last):
35	  File "/opt/pyp/bin/run/pyp", line 3992, in <module>
36	    particle_cspt.merge_movie_files_in_job_arr(
37	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
38	    return func(*args, **kwargs)
39	  File "/opt/pyp/src/pyp/refine/csp/particle_cspt.py", line 380, in merge_movie_files_in_job_arr
40	    with open(movie_file) as f:
41	FileNotFoundError: [Errno 2] No such file or directory: 'stacks.txt'
42	2024-02-08 12:09:42 [ERROR] /opt/pyp/bin/run/pyp:4012 | PYP (csp_local_merge) failed
43	2024-02-08 12:09:43 [INFO] /opt/pyp/bin/run/pyp:4018 | Deleted temporary files from /scratch/project_2009485/nextPYP/tmp/huiskone/3185360_1

I have given the original_volumes.txt file instead of the par.bz2 file, as follows

Any suggestions how to proceed? I have tried rerunning the preprocessing.

[yezhou867]

https://github.com/orgs/nextpyp/discussions/57#discussioncomment-8361155

Should be the other way to specify the input parameters.