creating spk files

[hamid13r]

Hi,

I am trying to run nextpyp on manually picked particles. However, I cannot get the program to read .box files. the spk files that are provided do successfully import and I believe they include initial orientation.

Does anyone know how to convert manual picks (with orientations) to spk files? It seems like they are binary files and I don't know how to generate them.

Best,
Hamid

[abartesaghi]

How are you manually picking your particles, inside nextPYP or outside? The .spk files are simply IMOD models that can be created using 3dmod. If you have .box files, you may be able to use point2model to convert them into IMOD format (assuming your particles where picked on tomograms produced by nextPYP). Just make sure you save them using the .spk extension. I don't think we support orientations when importing .spk coordinates. If you have external coordinates with orientations, you may need to provide that information separately. The coordinates themselves using .spk files as described above, and the orientations using the same format used in the *_volumes.txt files produced by the pre-processing block in the frealign/ folder (using the columns normalX, normalY, and normalZ). The interesting part would be to figure out the proper Euler angle convention that nextPYP is expecting.

[JuhaHuiskonen]

Should the .spk files have the same root filename as the original tilt-series? Are contour ID (and model ID) in these files ignored?

[yezhou867]

Yes, the root name is the same as the tilt series root name. Contour IDs are ignored.

[hamid13r]

Thank you for the quick response. Our particles have been manually picked (using dipole picking) from before and I am hoping to test the classification/refinement capabilities for subtomogram averaging between nextPYP and warp/relion.

We do have some experience converting orientation (going between i3 and relion), but I will have to see the difference between our tomograms and nextPYP-generated tomograms as well.

is the *_volumes.txt read in the importing step or I need to add it to the scratch directory myself?

[yezhou867]

*_volumes.txt is generated by nexPYP when there are particles picked. In your case, these will be your imported particles but with no alignment information. Coordinates from the box file probably need to be in an X, Y, and Z (thickness) order. If you have a RELION version of your refinement, would be easier to use "import data (tomography from star)" to properly import alignment, the output is also a *_volumes.txt file, which should be in the main project folder if it runs successfully. Hope that will help you get the correct reconstruction from your previous alignment in nextPYP.

[hamid13r]

Thank you. That was going to be my next worry on keeping the tomogram coordinates identical. We don't use relion pipeline but I believe I can create those files for nextPYP.

[JuhaHuiskonen]

Hi @hamid13r - Have you been able to import your particle coordinates with orientations? You mentioned "dipole picking" so I assume you are using Dynamo. I have also Dynamo TBL file, and I have converted it to STAR format in the past. Just wondering if the format is exactly what nextPYP assumes. Is it enough to have just the particle coordinate tags and orientation tags?

[yezhou867]

Since the tilt alignment will affect the tomogram reconstructions which sometimes could show random differences, theimport from starwill also need the tomograms.star to get the same tomogram reconstruction in nextPYP to match the star coordinates.

[hamid13r]

We actually used i3. points2model in imod works for importing test particles, but I have not tested the workflow with our tomograms to see where we end up with the orientations. I will post here if I successfully made the importing work.

[JuhaHuiskonen]

I managed to import XYZ coordinates as follows

# convert  XYZ coordinates to an IMOD model.  Pix size and dimensions are read from the MRC file
for f in *.box; do point2model -in "$f" -im "${f%.box}.mrc" -out "${f%.box}.mod" -sphere 5; done

# flipping Y and Z coordinates is required. Change extension from mod to spk
for f in *.mod; do imodtrans -Y "$f" "${f%.mod}.spk"; done

[JuhaHuiskonen]

I wrote now some scripts to read in orientations from a STAR file and write them in the volumes.txt file which I am then giving as an input to a refinement block. I am testing this by just averaging the particles using the refinement run (skipping both refinement steps). However, I am getting an error that I have been unable to debug. It is always the first tilt series which fails with the following error. If I remove this tilt series, then the next one will fail, so it seems the error is linked to the first tilt series, no matter what the actual data in it is. Any idea what might cause this?

499	2024-02-18 20:30:00 [INFO] pyp/align/core.py:57 | CSP Total time elapsed: 00h 01m 28s
500	2024-02-18 20:30:00 [INFO] /opt/pyp/bin/run/pyp:57 | Total time elapsed (csp_swarm): 00h 05m 21s
501	Traceback (most recent call last):
502	  File "/opt/pyp/bin/run/pyp", line 3914, in <module>
503	    csp_swarm(args.file, parameters, int(args.iter), args.skip, args.debug)
504	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
505	    return func(*args, **kwargs)
506	  File "/opt/pyp/bin/run/pyp", line 2456, in csp_swarm
507	    align.csp_refinement(
508	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
509	    return func(*args, **kwargs)
510	  File "/opt/pyp/src/pyp/align/core.py", line 1676, in csp_refinement
511	    new_par_file = csp_run_refinement(
512	  File "/opt/pyp/src/pyp/align/core.py", line 1222, in csp_run_refinement
513	    parx_object.data = merge_parx_particle_cspt(
514	  File "/opt/pyp/src/pyp/refine/csp/particle_cspt.py", line 186, in merge_parx_particle_cspt
515	    return frealign_parfile.Parameters.csp_merge_parameters_frame_refinement(
516	  File "/opt/pyp/src/pyp/inout/metadata/frealign_parfile.py", line 943, in csp_merge_parameters_frame_refinement
517	    par_array_list.append(Parameters.from_file(parx).data)
518	  File "/opt/pyp/src/pyp/inout/metadata/frealign_parfile.py", line 1447, in from_file
519	    data, prologue, epilogue = cls.columns_from_parfile(
520	  File "/opt/pyp/src/pyp/inout/metadata/frealign_parfile.py", line 1525, in columns_from_parfile
521	    assert (not cls.has_nan(data)), "Your parfile has missing values NaN. Please check!"
522	AssertionError: Your parfile has missing values NaN. Please check!
523	2024-02-18 20:30:00 [ERROR] /opt/pyp/bin/run/pyp:3927 | PYP (cspswarm) failed
524	2024-02-18 20:30:05 [INFO] /opt/pyp/bin/run/pyp:3273 | Job 3233163_13 (v0.6.2) launching on c1141.mahti.csc.fi using 128 task(s) and 220 GB of RAM
525	2024-02-18 20:30:05 [INFO] pyp/refine/csp/particle_cspt.py:57 | Total time elapsed (csp_local_merge): 00h 00m 00s
526	Traceback (most recent call last):
527	  File "/opt/pyp/bin/run/pyp", line 3992, in <module>
528	    particle_cspt.merge_movie_files_in_job_arr(
529	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
530	    return func(*args, **kwargs)
531	  File "/opt/pyp/src/pyp/refine/csp/particle_cspt.py", line 380, in merge_movie_files_in_job_arr
532	    with open(movie_file) as f:
533	FileNotFoundError: [Errno 2] No such file or directory: 'stacks.txt'
534	2024-02-18 20:30:05 [ERROR] /opt/pyp/bin/run/pyp:4012 | PYP (csp_local_merge) failed
535	2024-02-18 20:30:06 [INFO] /opt/pyp/bin/run/pyp:4018 | Deleted temporary files from /scratch/project_2009485/nextPYP/tmp/huiskone/3233163_13

[yezhou867]

Something happens during the conversion and the parameters files have Nan values exist. It's best to use nextPYP's import from RELION job to do this. We haven't released any info about the volumes.txt file convention, it's not straightforward to do this outside nextPYP. Prepare RELION's convention first (tomograms.star, .tlt and .xf files), then importing the project will be much easier I think.

[JuhaHuiskonen]

I convert the angles from STAR to Frealign convention and insert the values in the *_volumes.txt files produced by the pre-processing block in the frealign/ folder (using the columns normalX, normalY, and normalZ) as suggested above. I can now confirm that actually this works for me if I carry out the entire workflow on the same machine (I have picked a lipid bilayer in Dynamo, converted the Dynamo TBL file to STAR with dynamo2warp and then extracted the angles. When I average these particles in nextPYP, I get a lipid bilayer suggesting that the angles are correct). Happy to provide the script if someone wants to try.

But because of the IMOD patch tracking segfault issue #64, I ended up starting the workflow on one machine and then continuing the refinement on a different machine. Something goes wrong when I transfer the files and seems to lead to this issue. If #64 can be solved then I think importing coordinates and Euler angles would work.

[JuhaHuiskonen]

Further tests showed that these angles were not correct (initially I had forgotten to tick on "skip refinement" so the imported angles had been refined). I will report here if I figure out the angle convention.