nextPYP
Merge task failed #111
-
|
hello, Someone knew what the problem was? |
Beta Was this translation helpful? Give feedback.
Replies: 10 comments 6 replies
-
|
How many particles do you have in these tilt-series? The logs seem to indicate that no particles were actually extracted. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your reply,
The software extracted 90000 particles in preprocces step, but I don't know
if this extraction is real.
…On Mon, May 20, 2024, 23:24 Alberto Bartesaghi ***@***.***> wrote:
How many particles do you have in these tilt-series? The logs seem to
indicate that no particles were actually extracted.
—
Reply to this email directly, view it on GitHub
<#111 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BA4AUCSJISIQPBDVNNR6LO3ZDJLWJAVCNFSM6AAAAABH7CNCI2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKMBQGU4TO>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
OK, do you see any blue circles in the Reconstruction tab (pre-processing block)? And, where is the 90k number reported? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I see the blue circles.
The 90k particles report in the end of particles detection before particles
renfiment.
…On Tue, May 21, 2024, 00:43 Alberto Bartesaghi ***@***.***> wrote:
OK, do you see any blue circles in the Reconstruction tab (pre-processing
block)? And, where is the 90k number reported?
—
Reply to this email directly, view it on GitHub
<#111 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BA4AUCWVP6UXKCEH3JACL6DZDJVADAVCNFSM6AAAAABH7CNCI2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKMBRGE4DC>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Can you share the contents of the |
Beta Was this translation helpful? Give feedback.
-
|
Hi Alberto,
Sorry for the late response,
Because this step failed, I removed it. I will rerun it and send you the commands.
Thank you.
Ariel Cohen
MSc student in Physical Chemistry at Raviv lab
Los Angeles building, room 23.
Phone Number- +972523969740
[cid:231f8988-66ba-4216-98f5-513cc72a2c78]
…________________________________
From: Alberto Bartesaghi ***@***.***>
Sent: Tuesday, May 21, 2024 6:22 AM
To: nextpyp/.github ***@***.***>
Cc: ariel234156 ***@***.***>; Author ***@***.***>
Subject: Re: [nextpyp/.github] Merge task failed (Discussion #111)
Can you share the contents of the Commands field at the top the logs for the corresponding Launch job?
—
Reply to this email directly, view it on GitHub<#111 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BA4AUCR6PUCWGVUJRDUKMZDZDK4ZHAVCNFSM6AAAAABH7CNCI2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKMBTGAYTK>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Alberto, i attached the commands from particle refinement process:
"/opt/pyp/bin/run/csp" -data_parent "/mnt/nvme1n1/users/coari/projects/ITT-KABY/tomo-preprocessing-fAqQeIsEqhk6MI0Z" -particle_mw 55.0 -particle_rad 50.0 -particle_sym "C14" -extract_box 256 -extract_bin 2 -extract_fmt frealign -refine_model "/mnt/nvme1n1/users/coari/YK_20240402_154016/outputfile.mrc" -refine_parfile_tomo "/mnt/nvme1n1/users/coari/projects/ITT-KABY/tomo-preprocessing-fAqQeIsEqhk6MI0Z/frealign/20240521_180038_tomo-preprocessing-fAqQeIsEqhk6MI0Z_original_volumes.txt" -refine_rhref "8" -refine_fboost -refine_skip -reconstruct_maxtilt 60.0 -reconstruct_mintilt -60.0 -csp_UseImagesForRefinementMax 10 -csp_NumberOfRandomIterations 50000 -csp_refine_particles -csp_ToleranceParticlesPhi 10.0 -csp_ToleranceParticlesTheta 10.0 -csp_ToleranceParticlesShifts 50.0 -slurm_tasks 50 -slurm_memory 400 -slurm_walltime "9:00:00" -slurm_merge_tasks 50 -slurm_merge_memory 400 -slurm_merge_walltime "9:00:00"
Thank you.
Ariel Cohen
MSc student in Physical Chemistry at Raviv lab
Los Angeles building, room 23.
Phone Number- +972523969740
[cid:92b226a0-04f5-4de7-9c49-72ff1bd17209]
…________________________________
From: Ariel Cohen ***@***.***>
Sent: Tuesday, May 21, 2024 10:28 PM
To: nextpyp/.github ***@***.***>
Subject: Re: [nextpyp/.github] Merge task failed (Discussion #111)
Hi Alberto,
Sorry for the late response,
Because this step failed, I removed it. I will rerun it and send you the commands.
Thank you.
Ariel Cohen
MSc student in Physical Chemistry at Raviv lab
Los Angeles building, room 23.
Phone Number- +972523969740
[cid:231f8988-66ba-4216-98f5-513cc72a2c78]
________________________________
From: Alberto Bartesaghi ***@***.***>
Sent: Tuesday, May 21, 2024 6:22 AM
To: nextpyp/.github ***@***.***>
Cc: ariel234156 ***@***.***>; Author ***@***.***>
Subject: Re: [nextpyp/.github] Merge task failed (Discussion #111)
Can you share the contents of the Commands field at the top the logs for the corresponding Launch job?
—
Reply to this email directly, view it on GitHub<#111 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BA4AUCR6PUCWGVUJRDUKMZDZDK4ZHAVCNFSM6AAAAABH7CNCI2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKMBTGAYTK>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
The particles are definitely not being extracted for some reason. Here are some things to check:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Alberto,
Thank you for the quick response.
1.
The file that you mention isn't empty.
2.
I will do "verbose" and send you the logs of error code.
3.
I saw the blue circle in 3D after particle picking in the preprocessing in the right place.
Because the 90k particle creates 7Tb of data, I tried to use it in the smaller dataset, and I got the same problem.
Ariel Cohen
MSc student in Physical Chemistry at Raviv lab
Los Angeles building, room 23.
Phone Number- +972523969740
[cid:07ed4f9f-1b6f-45e2-8b79-5f8e0f4a87d8]
…________________________________
From: Alberto Bartesaghi ***@***.***>
Sent: Thursday, May 23, 2024 4:22 AM
To: nextpyp/.github ***@***.***>
Cc: ariel234156 ***@***.***>; Author ***@***.***>
Subject: Re: [nextpyp/.github] Merge task failed (Discussion #111)
The particles are definitely not being extracted for some reason. Here are some things to check:
* Can you confirm that the following file is not empty:
/mnt/nvme1n1/users/coari/projects/ITT-KABY/tomo-preprocessing-fAqQeIsEqhk6MI0Z/frealign/20240521_180038_tomo-preprocessing-fAqQeIsEqhk6MI0Z_original_volumes.txt?
* Try re-running the refinement block in verbose mode by checking the Verbose mode box in the Resources tab and seeing if there are any other warning messages.
* It looks like you are using geometry-based picking, is the 90k number reported inside the pre-processing block after you run particle picking?
* Also, I assume the blue circles are showing up in the right places in 3D, can you confirm this?
—
Reply to this email directly, view it on GitHub<#111 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BA4AUCS4D5LVI2PKAB5LDU3ZDVADRAVCNFSM6AAAAABH7CNCI2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKMRYGU3TA>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I attached the log from with verbose option. |
Beta Was this translation helpful? Give feedback.
-
|
This log shows that there were 11 particle projections extracted, which is different from before. Also, since this is a re-run, it just retrieved the metadata from the previous run so the logs won't show if there was a problem during extraction. You will need to clear the block by selecting |
Beta Was this translation helpful? Give feedback.
-
|
Hi, /bin/bash -c "cd '/mnt/nvme1n1/users/coari/projects/ITT-KABY/tomo-coarse-refinement-fBCtfXeuXI7Kei2l/swarm'; export cspswarm=cspswarm; /opt/pyp/bin/run/pyp --file Position_12_2 --iter 2 --no-skip --no-debug 2>&1 | tee ../log/Position_12_2_csp.log"
finally i add picture of the project- |
Beta Was this translation helpful? Give feedback.
-
|
It seems to be working now! You just need to increase the |
Beta Was this translation helpful? Give feedback.
-
|
I tried to run the command in 48 hours, but I got this error again during the split part. So, I don't think the problem is the timing. ERROR: pyp exiting at 2024-05-27 16:02:36226 | Traceback (most recent call last): Thank you, I really appreciate your help. |
Beta Was this translation helpful? Give feedback.
-
|
This looks like an issue we saw before where multiple split jobs can interfere with each other, essentially deleting each other's files and causing errors. There is an advanced parameter in the Resources tab called |
Beta Was this translation helpful? Give feedback.
-
|
I changed the parameters as you suggested and now it's working, thank you for all your help @abartesaghi |
Beta Was this translation helpful? Give feedback.
This looks like an issue we saw before where multiple split jobs can interfere with each other, essentially deleting each other's files and causing errors. There is an advanced parameter in the Resources tab called
Timeout to delete zombie runs (minutes)which defaults to 600. If your runs are longer than that, you could increase that number and see if the problem goes away. Another thing to keep in mind is that you need to make sure the compute nodes have enough scratch space to save the necessary particle stacks. And if you have multiple tilt-series running on the same node at the same time, you will need to add their storage requirements and ensure that the node has enough capacity.