container method not working. #1640
Comments
alejoe91
added
the
container
|
Hi, did it work before? The problem seems to be that spikeinterface fails to install inside the container.. If you installed version Did you also updgrade singularity? Can you share the entire code to reproduice the issue? |
|
If I remember correctly, I had the same issue with an older version of Singularity. It disappeared when I installed the latest version3 |
|
hi Thanks all, I upgraded the singularity, but the problem is still there. absl-py @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/abslpy/absl-py-1.4.0 |
|
And here is the code python code. import spikeinterface as si from spikeinterface.preprocessing import (bandpass_filter, notch_filter, common_reference, import numpy as np import warnings def current_path(): os.chdir('/cluster/home/sachur/Overlap_project/SpkSorting/RawDate') num_channels = 64 base_folder = Path(".") stream_name='Record Node 106#Acquisition_Board-100.Rhythm Data' os.chdir('/cluster/home/sachur/Overlap_project/SpkSorting/RawDate') recording.annotate(is_filtered=False) channel_ids = recording.get_channel_ids() print(f'Channel ids: {channel_ids}') import probeinterface as pi manufacturer = 'cambridgeneurotech' probe = pi.get_probe(manufacturer, probe_name) #probe.wiring_to_device('cambridgeneurotech_mini-amp-64') connector J2 TOP] probe.set_device_channel_indices(mapping_to_device) pi.write_prb(f"{probe_name}.prb",probegroup,group_mode="by_shank") recording_prb=recording print(f'Original channels:{recording.get_channel_ids()}') #brain_area_property_values = ['CA1']*32 + ['CA3']32 recording_prb.get_property("quality") recording_cmr=recording_prb recording_cmr = common_reference(recording_f, fs = recording_cmr.get_sampling_frequency() recording_sub = recording_cmr preprocessed=current_file_name + "_" + "preprocessed" print(f'Cached channels ids: {recording_saved.get_channel_ids()}') recording_loaded = si.load_extractor(base_folder / preprocessed) print("Properties after adding custom properties:", list(recording_prb.get_property_keys())) print(recording_f) sorter_params = job_kwargs sorter_params = dict(n_jobs=12, sorting = ss.run_kilosort3(recording=recording_saved, |
|
Additionally this should be the original error, I installed cuda then it changed to error above. Traceback (most recent call last): |
|
The problem might be due to singularity is not excluding the home lib, thus the all the thing in the container was shaded by that. yet I don't have any solution to solve this. Would somebody have any idea? |
|
I see, I have mainly used Docker, but I think that @DradeAW had similar issues. Maybe he could help? |
|
Did anyone make any progress on this issue? I'm getting the exact same thing - trying to run any sorter in a singularity container returns a ModuleNotFoundError |
|
Can you paste the full error? |
|
@sachuriga283 looking again, it might be fixed with a simple: |
|
The fix for me was to install the latest version of Go (and not the one recommended in the documentation!) |
|
Thanks all, I will try and report back. |
|
Here's my full error: SpikeSortingError Traceback (most recent call last) File ~/local/mamba/envs/si_env/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:137, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, with_output, **sorter_params) File ~/local/mamba/envs/si_env/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:583, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, extra_requirements, **sorter_params) SpikeSortingError: Spike sorting in singularity failed with the following error: |
|
Thanks, so something is wrong when it attempts to install spikeinterface in the container. What is the output before the error? Also, what version of spikeinterface and singularity are you using? We had some troubles with older versions of singularity |
|
Did you set the environment variable I had the same problem until I set that variable to point to the repo :) |
|
output before the error: Starting container I'm also running spikeinterface==0.97.1 on my local machine in a conda environment Singularity version: apptainer version 1.1.7-1.el8 I appreciate all the help and the quick replies. I'm in an institution where I can't control the version of singularity being used so if that is the issue I will need to chat to my sysadmin. |
|
I think you have the wrong version of Singularity. Here is my output: $ singularity --version
singularity-ce version 3.11.1 |
|
I believe sylabs' singularity and the linux foundation's apptainer are meant to be fully compatible |
|
To follow this up: I've spent quite a while troubleshooting this now and have found a couple sources for the error. If I enter the container myself and run: 'pip install --upgrade --no-input spikeinterface[full]==0.97.1' I get: 'ERROR: Could not install packages due to an EnvironmentError: [Errno 30] Read-only file system' So it definitely seems like it's failing to install spikeinterface from within the container. I will chat to my sysadmin about our version of singularity and how to allow the container to install software. Secondly I previously had an issue where the container was finding my local installs of python in my home directory, and I think it was confusing this with its own python installation. I had some conda environments stored under $HOME/local, and changed this to $HOME/.local, and it doesn't seem to find them under the hidden directory. Thought I'd mention this just so people know having conda environments stored like this could cause the container to not use its own python. An honest question here, just as I'm fairly new to using containers: my understanding is that containers should include all the necessary dependencies for an application pre-installed and ready to run. Why is it then that all the spikeinterface sorting containers on dockerhub don't include spikeinterface preinstalled? Would it be that tricky to have si_env included in the image? Is this so that the latest updates can be installed without having to reupload the docker image? Forgive me for the newbie question, it just seems to me that having a container ready to go as is is kind of the purpose of using containers in the first place. |
Very honest question! Yes indeed the reason is to decouple the development of individual sorters (each image should "fix" a sorter) to the development of SpikeInterface. SpikeInterface is still slowly changing, so if we installed it directly in the images we would need to update them quite often. Instead we prefer to install the current SI version in the container. |
|
That makes sense thanks for explaining! Just wanted to add for now I'm using the following workaround: I'm building the container using apptainer with the following .def file (substitute any sorter): This works well and I guess you could just update the definition and rebuild whenever you want a new version of spikeinterface. Hope this helps anyone else who runs into the same issue. |
|
Yep that sounds totally reasonable! Note that klusta unfortunately doesn't work with SI>=0.96 because it requires python 3.7! (I see you figured it out, just wanted to explain why ;)) |
Hi,
The code are running well till today, but after I updaeted to 0.97.1version suddenly apears the error below, does anyone know what cause the problem?
Sampling frequency (Hz): 30000.0
Singularity: pulling image spikeinterface/kilosort3-compiled-base
singularity pull --name kilosort3-compiled-base.sif docker://spikeinterface/kilosort3-compiled-base
�[33mWARNING:�[0m DEPRECATED USAGE: Environment variable SINGULARITY_PULLFOLDER will not be supported in the future, use APPTAINER_PULLFOLDER instead
INFO: Using cached SIF image
�[33mWARNING:�[0m DEPRECATED USAGE: Environment variable SINGULARITY_PULLFOLDER will not be supported in the future, use APPTAINER_PULLFOLDER instead
INFO: Using cached SIF image
kilosort3-compiled-base.sif
Starting container
Installing spikeinterface==0.97.0 in spikeinterface/kilosort3-compiled-base
Installing extra requirements: ['neo']
Running kilosort3 sorter inside spikeinterface/kilosort3-compiled-base
Stopping container
Traceback (most recent call last):
File "/cluster/home/sachur/Overlap_project/code/Python/OpenEphysForm/ks3_f_probe_20230508_Backup.py", line 148, in
sorting = ss.run_kilosort3(
File "/cluster/home/sachur/.local/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 691, in run_kilosort3
return run_sorter('kilosort3', *args, **kwargs)
File "/cluster/home/sachur/.local/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 137, in run_sorter
return run_sorter_container(
File "/cluster/home/sachur/.local/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 578, in run_sorter_container
raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting in singularity failed with the following error:
Traceback (most recent call last):
File "/cluster/home/sachur/Overlap_project/SpkSorting/RawDate/in_container_sorter_script.py", line 3, in
from spikeinterface import load_extractor
ModuleNotFoundError: No module named 'spikeinterface'
The text was updated successfully, but these errors were encountered: