Does nextPYP support to import eer(Electron-event representation) format file?

[zhang377]

Dear professor,
I met one question that I can't import tomography from raw data,here are errors:

5	Traceback (most recent call last):
6	File "/opt/pyp/bin/run/pyp", line 4100, in
7	x, y, z = get_image_dimensions(image_file)
8	File "/opt/pyp/src/pyp/inout/image/core.py", line 871, in get_image_dimensions
9	return [int(num) for num in output.split()]
10	File "/opt/pyp/src/pyp/inout/image/core.py", line 871, in
11	return [int(num) for num in output.split()]
12	ValueError: invalid literal for int() with base 10: 'ERROR:'
13	2024-10-08 14:25:23 [ERROR] /opt/pyp/bin/run/pyp:4126 | PYP (pypgain) failed

[zhang377]

I am slightly confused because the tutorial mentions importing TIFF format raw data, but I do not have any TIFF files in my folder. Could you please advise which files I should select for import? Here is the structure of my folder:

[abartesaghi]

Yes, nextPYP can read data in eer format.

Is the error you are seeing from the Logs tab or the Stream tab? The Logs tab is usually more helpful when errors like these occur. It will also be helpful to see the contents of the > Commands section at the top of the log window.

The Location to import your data should be the frames/ folder, and the wildcard *.eer.

[zhang377]

Thanks for your kind of help! I have changed the input file path as you advised,but there are still bugs.
here are commands:

"/opt/pyp/bin/run/gyp" -scope_pixel 1.9 -scope_voltage 300.0 -scope_tilt_axis -174.1 -data_path "/data/Softwares/nextpyp/shared/users/admin/projects/empire-11830-Position3-3yjSayZI2lpisy1LJEh5bgINjN0xVr/01082023_BrnoKrios_Arctis_WebUI_Position_3_raw/frames/*.eer" -data_path_mdoc "/data/Softwares/nextpyp/shared/users/admin/projects/empire-11830-Position3-3yjSayZI2lpisy1LJEh5bgINjN0xVr/01082023_BrnoKrios_Arctis_WebUI_Position_3_raw/*.mdoc" -data_mode tomo

here are Logs:

1 | 2024-10-08 21:56:41 [INFO] /opt/pyp/bin/run/pyp:3232 \| nextPYP v0.6.5 launching job 995404129501479419 on chenyf-S130 using 2 task(s)
-- | --
2 | 2024-10-08 21:56:41 [INFO] /opt/pyp/bin/run/pyp:380 \| Attempting to load existing parameters and metadata from /data/Softwares/nextpyp/shared/users/admin/projects/empire-11830-Position3-3yjSayZI2lpisy1LJEh5bgINjN0xVr/01082023_BrnoKrios_Arctis_WebUI_Position_3_raw/frames
3 | 2024-10-08 21:56:41 [INFO] /opt/pyp/bin/run/pyp:422 \| /data/Softwares/nextpyp/shared/users/admin/projects/empire-11830-Position3-3yjSayZI2lpisy1LJEh5bgINjN0xVr/01082023_BrnoKrios_Arctis_WebUI_Position_3_raw/frames/*.eer found 41 files to link into raw folder
4 | 2024-10-08 21:56:41 [INFO] /opt/pyp/bin/run/pyp:4098 \| Selecting image for preview: /data/Softwares/nextpyp/shared/users/admin/projects/empire-11830-Position3-3yjSayZI2lpisy1LJEh5bgINjN0xVr/01082023_BrnoKrios_Arctis_WebUI_Position_3_raw/frames/Position_3_040_-75.01_20230801_185628_EER.eer
5 | 2024-10-08 21:56:41 [INFO] /opt/pyp/bin/run/pyp:3178 \| Filesystem      Size  Used Avail Use% Mounted on
6 | 2024-10-08 21:56:41 [INFO] /opt/pyp/bin/run/pyp:3178 \| /dev/sdb3       117G   35G   77G  32% /tmp/pyp
7 | Traceback (most recent call last):
8 | File "/opt/pyp/bin/run/pyp", line 4100, in <module>
9 | x, y, z = get_image_dimensions(image_file)
10 | File "/opt/pyp/src/pyp/inout/image/core.py", line 871, in get_image_dimensions
11 | return [int(num) for num in output.split()]
12 | File "/opt/pyp/src/pyp/inout/image/core.py", line 871, in <listcomp>
13 | return [int(num) for num in output.split()]
14 | ValueError: invalid literal for int() with base 10: 'ERROR:'
15 | 2024-10-08 21:56:41 [ERROR] /opt/pyp/bin/run/pyp:4126 \| PYP (pypgain) failed

[abartesaghi]

nextPYP simply calls IMOD's header command to get the image dimensions, but that seems to be failing on your files for some reason. If you have IMOD installed in your system, what output do you get from running this command?

header -size /data/Softwares/nextpyp/shared/users/admin/projects/empire-11830-Position3-3yjSayZI2lpisy1LJEh5bgINjN0xVr/01082023_BrnoKrios_Arctis_WebUI_Position_3_raw/frames/Position_3_040_-75.01_20230801_185628_EER.eer

If you don't have IMOD installed, I can explain how to do this using nextPYP's container.

[zhang377]

Thank you for your patience!
Currently, I have only installed nextPYP on my machine, and I haven't installed imod yet.
To use nextPYP effectively, do I need to install imod as well?
Also, I'm curious as to why there were no issues when I imported the dataset from the tutorial. Could you explain that?

[abartesaghi]

You don't need to install IMOD to run nextPYP. IMOD's header command is just a handy way to check the integrity of the data.
Here is how to run header using the nextPYP's container. First, cd to nextPYP's installation folder, then run:

singularity run -B /data pyp.sif header -size /data/Softwares/nextpyp/shared/users/admin/projects/empire-11830-Position3-3yjSayZI2lpisy1LJEh5bgINjN0xVr/01082023_BrnoKrios_Arctis_WebUI_Position_3_raw/frames/Position_3_040_-75.01_20230801_185628_EER.eer

The tutorial uses data in *.tif format, not *.eer, so you won't see errors there.

[zhang377]

The output is 16384 16384 728 after excuted the header command.
At Raw data,I imported *.eer file and .mdoc file, I also set the parameters in Microscopy parameters,but there is nothing excuted and no logs.It just keep this state as picture shows(No matter there is .mdocfile,the state is same as picture，I didn't change gain reference and resources).

[zhang377]

when i put my mouse at this area for a while,it shows "the block is waiting to run",I really don't know how to resolve this tricky problem and i have been stuck at this for three days ,i genuinelly hope you could help me with this.

[abartesaghi]

Your .*eer file looks fine since header is able to get the size information just fine. The operation inside nextPYP is identical to the command you ran, so I don't have an explanation for why this is failing. Can you confirm that /data is in the list of binds in the configuration file?

As far the jobs not starting, you can look at the file local/logs/micromon as explained in the Troubleshooting section of the docs. One possibility is that other jobs may be taking up resources and preventing any new jobs from running, do you have any older jobs that show up in the running state in the Jobs panel? If so, try canceling them using the red button. In some rare cases, the status of the resource queue can become out of sync. If this happens, restarting nextPYP should fix the issue.

[zhang377]

Professor, i tackled the problem that jobs can't starting by restart nextPYP as you recommended.

sorry,i didn't get the meaning of " /data is in the list of binds in the configuration file",what should i do to inspect it?Could you please explain that?

now,the big question that i can't import raw data is this：
Hope to get your kindly help.

1	2024-10-10 21:15:10 [INFO] /opt/pyp/bin/run/pyp:3232 | nextPYP v0.6.5 launching job 2085488363668997365 on chenyf-S130 using 1 task(s)
2	2024-10-10 21:15:10 [INFO] /opt/pyp/bin/run/pyp:4098 | Selecting image for preview: /data/zchSoftwares/nextpyp/shared/users/admin/projects/test-289lyu/archive/11830/data/chlamy_visual_proteomics/01122021_BrnoKrios_arctis_lam1_pos3/frames/2021201_BrnoKrios_arctis_lam1_pos3_032_EER.eer
3	2024-10-10 21:15:10 [INFO] /opt/pyp/bin/run/pyp:3178 | Filesystem Size Used Avail Use% Mounted on
4	2024-10-10 21:15:10 [INFO] /opt/pyp/bin/run/pyp:3178 | /dev/sdb3 117G 35G 76G 32% /tmp/pyp
5	Traceback (most recent call last):
6	File "/opt/pyp/bin/run/pyp", line 4100, in
7	x, y, z = get_image_dimensions(image_file)
8	File "/opt/pyp/src/pyp/inout/image/core.py", line 871, in get_image_dimensions
9	return [int(num) for num in output.split()]
10	File "/opt/pyp/src/pyp/inout/image/core.py", line 871, in
11	return [int(num) for num in output.split()]
12	ValueError: invalid literal for int() with base 10: 'ERROR:'
13	2024-10-10 21:15:10 [ERROR] /opt/pyp/bin/run/pyp:4126 | PYP (pypgain) failed

[abartesaghi]

Can you share the contents of your config.toml file? This file will be in the folder where you installed nextPYP.

[zhang377]

Professor ，here are the contents of config.toml file:

[pyp]
# path to the PYP container file, from a SLURM compute node's point of view
container = '/data/zchSoftwares/nextpyp/pyp.sif'

# fast storage for temporary files, ideally local to the compute node
scratch = '/tmp/pyp'


[web]

# storage space for website files, database, etc
# for best performance, this should be on a filesystem local to the web server
localDir = '/data/zchSoftwares/nextpyp/local'

# storage space for files shared between the web server and the SLURM cluster
# this area should have a lot of available space for large files
sharedDir = '/data/zchSoftwares/nextpyp/shared'

# directories for workflow files
workflowDirs = ['/data/zchSoftwares/nextpyp/workflows']

# authentication mode: how to log into the website
# disable authentication entirely in single-user mode
auth = "none"
host = '0.0.0.0'

[abartesaghi]

Please add the following line to the [pyp] section:

binds = [ '/data' ]

Then restart nextPYP and try to run the Data Import block again.

There is a description for why you need to do this in the Documentation.

[zhang377]

Thank you professor! I have done what you recommended,still, there is the same bug,it seems like that there is no connection between config.toml and this bug.
here are all informations:

[abartesaghi]

can you confirm that you restarted nextPYP after making the change to the config.toml file?

[zhang377]

Yes,Professor.I indeed restarted nextPYP after making the change to the config.toml file.At first ,my data is in this installing path:/data/zchSoftwares/nextpyp/shared/users/admin/projects/archive/11830/data/chlamy_visual_proteomics/01122021_BrnoKrios_arctis_lam1_pos3/frames,i changed the configure file and restarted nextPYP,it show bugs as before.then I move my data at /data/archive/11830/data/chlamy_visual_proteomics/01122021_BrnoKrios_arctis_lam1_pos3/frames path,yet,there is the same bug.when i installed nextPYP,i only changed the installing path compared to tutorial ,from /opt/nextPYP to/data/zchSoftwares/nextpyp/.
Professor,do you think I need to reinstall nextPYP? If i should reinstall nextPYP,what should i do to remove nextPYP totally? Also,i found the download speed is very slow in Asia,could you give me some advise to download it more easily?

[abartesaghi]

There is no need to reinstall nextPYP (specially if you have a slow internet connection). Can you share the contents of the log file: /data/zchSoftwares/nextpyp/local/logs/micromon? (I think that is the correct location for your setup, if not, just look for the filelocal/logs/micromon under your nextPYP installation directory).

[zhang377]

Thank you , professor!
I uploaded the micromon file.To upload sucessfully,I renamed micromon as micromon.txt
micromon.txt

[zhang377]

professor,here is my configuration:

[abartesaghi]

Everything looks fine as far as I can tell. Any chance the *.eer files are on a different file system than /data? For example, maybe they are symlinks under /data but that point to another location in the file system? If that's the case, that location will also need to be added to the list of binds.
One other possibility is that the import task could be running out of memory. If that's the case, you can try increasing the value of the parameter Memory (launch task) in the Resources tab.

[zhang377]

Thanks for your patience,professor!
I will change an machine to run nextPYP,maybe there are some bugs in this machine itself.

[abartesaghi]

Still, I will consider trying the last two suggestions because there is no reason why it shouldn't work. Also, the next release will have better error reporting to help troubleshoot cases like this.

[zhang377]

Thank you, Professor!
I’m a big fan of NextPYP and genuinely looking forward to the next release!