New container relion 4.0.1.sm61 #620
Comments
|
@vennand - could you test this container and see if it all works as expected? |
|
How do I transfer data to the neurodesktop to test? It opens as expected, but I need to launch a job to know if it'll work. |
|
Are you running neurodesktop locally in docker? If yes, you have a shared directory between the desktop and the host. Alternatively you can drag and drop files on the desktop and guacamole will upload the file (has to be one file, can't be a directory) |
|
I'm trying locally in docker, and I just noticed the directory, thanks! Is it possible to do a GPU passthrough with the local docker? I'm pretty sure I won't be able to test if the GPU settings work otherwise. Though so far, there was no error message saying it was CPU only. Though I'm not convinced it compiled with GPU support if the machine that built the container didn't have a GPU. With the new version of relion (ver5.0), they explicitly state that the compiler tries to detect a GPU, and if not, compiles for CPU only, even if the a GPU architecture is provided. |
|
Dear @vennand yes, you can pass your GPU into the docker container: to check if it worked, run nvidia-smi in the desktop container afterwards that would be annoying if it needs a GPU to compile. We do not have the ability to run a GPU node for building containers. |
|
We might just be limited to version 4 for now then. As far as I can tell, version 5 is still in beta, so it might not be advisable to use it for research anyway. I tried running your command, but I get the following error message Didn't find anything relevant with a very quick Google. Any idea what could cause this? Also, I won't be able to touch this until the 16th of April unfortunately, but I plan on getting back to it. |
|
Dear @vennand Did you install the nvidia-container-toolkit beforehand? |
|
I had not, but I get the same error after installing it |
|
what are you getting when you run nvidia-smi on your host system? |
|
`+---------------------------------------------------------------------------------------+ +---------------------------------------------------------------------------------------+ |
|
can you try this? https://www.howtogeek.com/devops/how-to-use-an-nvidia-gpu-with-docker-containers/ It needs a restart restart of the docker daemon and potentially apt-get install -y nvidia-docker2 |
|
I've installed nvidia-docker2, but I've also ran this I don't know which one worked, but it worked. I'll try to test it now, but I don't know if I'll have time |
|
Hi @stebo85, I've finished testing. Relion works as intended, but none of the jobs showed up when running "nvidia-smi", even though we could see the GPU being used. Not sure if that's an issue with the GPU passthrough, but it is using the GPU. Another important issue is that one of the third party software I install along with relion doesn't work. Basically, CTFFIND 4.1.14 fails if it's compiled with GCC 8 or above. The fix I've found is to modify the code, which doesn't seem practical or elegant to do in the neurodesk script. |
|
Dear @vennand, Fixing a software live for a container is a tricky one. I have done various things in the past depending on the project:
|
|
To test the GPU, I simply watched nvidia-smi I don't know exactly how the code accesses the GPU, but I can probably find out if that's relevant. When I run Regarding fixing the software, I think I'll go with option 2, since the source code is only 11MB. Do you want me to push the fix now, or should we investigate the GPU "issue" before? |
|
Interesting. I don't know what causes this behaviour, but I guess if it works it works no matter where the GPU tasks show up. Happy for you to push the fix now :) Let's see if we can get this work! |
|
@stebo85 Would you know what this error means? |
|
Yes, you need to update the GitHub url of neurodocker:
For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
[end of output]
Thank you
Steffen
…--
W: https://mri.sbollmann.net<https://mri.sbollmann.net/> | W: https://www.neurodesk.org<https://www.neurodesk.org/> | T: https://twitter.com/sbollmann_MRI | G: https://github.com/stebo85
Book meeting: https://calendly.com/s-bollmann/meeting
________________________________
From: Andre Venne ***@***.***>
Sent: Monday, May 27, 2024 9:15:30 AM
To: NeuroDesk/neurocontainers ***@***.***>
Cc: Steffen Bollmann ***@***.***>; Mention ***@***.***>
Subject: Re: [NeuroDesk/neurocontainers] New container relion 4.0.1.sm61 (Issue #620)
@stebo85<https://github.com/stebo85> Would you know what this error means?
$ bash build.sh -ds
Entering Debug mode
WARNING: Skipping neurodocker as it is not installed.
Defaulting to user installation because normal site-packages is not writeable
Collecting https://github.com/ReproNim/neurodocker/tarball/master
Downloading https://github.com/ReproNim/neurodocker/tarball/master
- 77.3 kB 10.0 MB/s 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [36 lines of output]
/tmp/pip-build-env-aeraeba5/overlay/local/lib/python3.10/dist-packages/setuptools_scm/_integration/setuptools.py:31: RuntimeWarning:
ERROR: setuptools==59.6.0 is used in combination with setuptools_scm>=8.x
Your build configuration is incomplete and previously worked by accident!
setuptools_scm requires setuptools>=61
Suggested workaround if applicable:
- migrating from the deprecated setup_requires mechanism to pep517/518
and using a pyproject.toml to declare build dependencies
which are reliably pre-installed before running the build tools
warnings.warn(
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-aeraeba5/overlay/local/lib/python3.10/dist-packages/hatchling/build.py", line 112, in prepare_metadata_for_build_wheel
directory = os.path.join(metadata_directory, f'{builder.artifact_project_id}.dist-info')
File "/tmp/pip-build-env-aeraeba5/overlay/local/lib/python3.10/dist-packages/hatchling/builders/wheel.py", line 825, in artifact_project_id
self.project_id
File "/tmp/pip-build-env-aeraeba5/overlay/local/lib/python3.10/dist-packages/hatchling/builders/plugin/interface.py", line 374, in project_id
self.__project_id = f'{self.normalize_file_name_component(self.metadata.core.name)}-{self.metadata.version}'
File "/tmp/pip-build-env-aeraeba5/overlay/local/lib/python3.10/dist-packages/hatchling/metadata/core.py", line 149, in version
self._version = self._get_version()
File "/tmp/pip-build-env-aeraeba5/overlay/local/lib/python3.10/dist-packages/hatchling/metadata/core.py", line 248, in _get_version
version = self.hatch.version.cached
File "/tmp/pip-build-env-aeraeba5/overlay/local/lib/python3.10/dist-packages/hatchling/metadata/core.py", line 1466, in cached
raise type(e)(message) from None
LookupError: Error getting the version from source `vcs`: setuptools-scm was unable to detect version for /tmp/pip-req-build-wu94yd8o.
Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.
For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
—
Reply to this email directly, view it on GitHub<#620 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA6V2W2Z6DSY4RYJNEGIPO3ZELMRFAVCNFSM6AAAAABE3BMHXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSG44TSMRSGQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hey, I'm back working on this. I'll start implementing the other software soon. But first, I tested this version of relion on our other GPUs, and it runs without issues. Perhaps the default setting (sm35) is too old, but this one works. I'm thinking it would be simpler for users to only package this one. |
|
Great to hear that Relion is working :) ok, makes sense that the newer version works better. CUDA is usually quite backwards compatible, so if you have fairly current driver versions that makes sense. Yes, put the version you found working best in the apps.json and this will trigger the release process. Thank you for getting this to work !!! |
There is a new container by @stebo85, use this command to test:
If test was successful, then add to apps.json to release:
https://github.com/NeuroDesk/neurocommand/edit/main/neurodesk/apps.json
Please close this issue when completed :)
The text was updated successfully, but these errors were encountered: