error when constrained refinement

[hustts]

Dear All, when I did the reference base refinement, it run well and can obtained reasonable resolutin, while when I click anything in constrained refinement (like refine particle alignment or refine CTF pre particle), it always give me some error that looks like it cannt get the meanscore. Any suggestion will be quite grateful!!! and my version is 0.6.5
error message：

82 | 2024-11-06 09:28:14 [INFO] pyp/refine/csp/particle_cspt.py:427 \| Merging parameter files
83 | 2024-11-06 09:28:14 [INFO] pyp/refine/csp/particle_cspt.py:488 \| Running reconstruction
84 | 2024-11-06 09:28:15 [INFO] pyp/refine/csp/particle_cspt.py:545 \| Skip modifying metadata for 1 film only
85 | 2024-11-06 09:28:16 [INFO] pyp/analysis/scores.py:57 \| Shaping scores took: 00h 00m 00s
86 | 2024-11-06 09:28:16 [INFO] pyp/refine/csp/particle_cspt.py:57 \| Running reconstruction took: 00h 00m 00s
87 | 2024-11-06 09:28:16 [INFO] pyp/refine/csp/particle_cspt.py:57 \| Total time elapsed (csp_local_merge): 00h 00m 01s
88 | 2024-11-06 09:28:16 [ERROR] /opt/pyp/bin/run/pyp:166 \|
89 | ERROR: pyp exiting at 2024-11-06 09:28:16
90 | Traceback (most recent call last):
91 | File "/opt/pyp/bin/run/pyp", line 3949, in <module>
92 | particle_cspt.merge_movie_files_in_job_arr(
93 | File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
94 | return func(*args, **kwargs)
95 | File "/opt/pyp/src/pyp/refine/csp/particle_cspt.py", line 547, in merge_movie_files_in_job_arr
96 | run_reconstruction(
97 | File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
98 | return func(*args, **kwargs)
99 | File "/opt/pyp/src/pyp/refine/csp/particle_cspt.py", line 789, in run_reconstruction
100 | par_obj = call_shape_phase_residuals(
101 | File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
102 | return func(*args, **kwargs)
103 | File "/opt/pyp/src/pyp/analysis/scores.py", line 824, in call_shape_phase_residuals
104 | par_obj = shape_phase_residuals(
105 | File "/opt/pyp/src/pyp/analysis/scores.py", line 436, in shape_phase_residuals
106 | thresholds[g, f] = np.sort(meanscore)[
**### 1. 107 | IndexError: index 0 is out of bounds for axis 0 with size 0**
108 |  
109 | During handling of the above exception, another exception occurred:
110 |  
111 | Traceback (most recent call last):
112 | File "/opt/pyp/bin/run/pyp", line 3969, in <module>
113 | trackback()
114 | File "/opt/pyp/bin/run/pyp", line 3164, in trackback
115 | get_free_space(Path(os.environ["PYP_SCRATCH"]).parents[0])
116 | File "/usr/local/envs/pyp/lib/python3.8/pathlib.py", line 621, in __getitem__
117 | raise IndexError(idx)
118 | IndexError: 0
119 |  
120 | During handling of the above exception, another exception occurred:
121 |  
122 | Traceback (most recent call last):
123 | File "/opt/pyp/bin/run/pyp", line 4591, in <module>
124 | trackback()
125 | File "/opt/pyp/bin/run/pyp", line 3164, in trackback
126 | get_free_space(Path(os.environ["PYP_SCRATCH"]).parents[0])
127 | File "/usr/local/envs/pyp/lib/python3.8/pathlib.py", line 621, in __getitem__
128 | raise IndexError(idx)
129 | IndexError: 0

[abartesaghi]

This error may occur when there are very few particle projections extracted from a tilt-series. The fix will be part of the next release, but in the meantime, try using a Bundle size > 1 (Resources tab) and let us know if that fixes the problem.

[hustts]

Dear Abartesaghi,

Thank you so much for this helpful suggestion! when I increase bundle size = 4, there is no more error about meanscore. for the iteration 2 (split) go well, but for the next step -iteration 2 (merge), and it took a long time and finally Job reached walltime, is it normal?

As I just have few hundred particles, and the reference base refinement give me a good result, can I trust it (how to tell if it is model bias)?

Thank you!
Best Regards
Bo

[abartesaghi]

I'm confused. If the merge job reached the walltime, that usually means that the reconstruction failed, but it looks like you did get a map in the end?

There merge job could reach the walltime when:

1. The split jobs take too long to run eventually causing the merge job to timeout
2. Some of the split jobs may have failed and the merge job is unaware of this

In the first case, increasing the walltime for the merge job should solve the problem. In the second case, one would need to figure out why some of the split jobs failed in the first place.

If you were indeed able to get a reconstruction and you are trying to determine whether model bias is present, there a couple of things to consider:

1. If the FSC falls off sharply at the max resolution used for refinement, that's usually a sign of overfitting. Try using a lower value for the maximum resolution and make sure the size of the search space (Euler angles and translation range) are appropriate.
2. To confirm that particles are being aligned correctly, you can visualize the assigned orientations and translations on the original tomograms using the ChimeraX/ArtiaX display. If the majority of particles are misaligned, you may need to re-run refinement using different parameters.