SIPEC on HPC without root using conda

[jthiels]

Hi,

Users of large HPC clusters normally don't have sudo or root permissions. They cannot use Docker for security reasons, and the Singularity build of the Docker container fails because of the need to run as root. I was able to install SIPEC for the users of our HPC cluster inside a conda environment that did not require root. I'm attaching the environment.yml file as a .txt file so that anyone can install it with the extra instructions below.

After creating the conda environment with the environment.yml file , I had to manually install (conda install -c conda-forge) cudnn==8.0.0 and cudatoolkit==11.0.3 before installing cudatoolkit-dev==11.0.3 to get the cuda tools with the correct versions. I also had to copy the SwissKnife functions into the relevant conda site-packages location so that they could be imported by Python. In my environment it was ~/miniconda3/envs/sipec/lib/python3.7/SwissKnife. Once I did that, the software worked on the GPU with no apparent issues reported by the user.

The current setup file in SIPEC inside the repository requires a Ubuntu (or at least a Debian-based OS) that can install the Python versions from deadsnakes and libraries that we could not get installed on the cluster for being quite old or having security issues. If you have time, it'd be great to see the code updated with more recent versions of Python, cuda, cudnn, tensorflow, etc. Your software has features the other similar pose estimation software does not, according to the researcher here, so it's really great for them and we'd like to continue supporting it. Thanks for your hard work.

environment.txt.txt

Best,
John

[smutrejaUC]

Hi,

Users of large HPC clusters normally don't have sudo or root permissions. They cannot use Docker for security reasons, and the Singularity build of the Docker container fails because of the need to run as root. I was able to install SIPEC for the users of our HPC cluster inside a conda environment that did not require root. I'm attaching the environment.yml file as a .txt file so that anyone can install it with the extra instructions below.

After creating the conda environment with the environment.yml file , I had to manually install (conda install -c conda-forge) cudnn==8.0.0 and cudatoolkit==11.0.3 before installing cudatoolkit-dev==11.0.3 to get the cuda tools with the correct versions. I also had to copy the SwissKnife functions into the relevant conda site-packages location so that they could be imported by Python. In my environment it was ~/miniconda3/envs/sipec/lib/python3.7/SwissKnife. Once I did that, the software worked on the GPU with no apparent issues reported by the user.

The current setup file in SIPEC inside the repository requires a Ubuntu (or at least a Debian-based OS) that can install the Python versions from deadsnakes and libraries that we could not get installed on the cluster for being quite old or having security issues. If you have time, it'd be great to see the code updated with more recent versions of Python, cuda, cudnn, tensorflow, etc. Your software has features the other similar pose estimation software does not, according to the researcher here, so it's really great for them and we'd like to continue supporting it. Thanks for your hard work.

environment.txt.txt

Best, John

Hi John,

Thanks so much for the steps you outlined and the life saver yaml file. I have reached the final step. However, as I am testing the segmentation.py file I am not sure of all the paths that need to be set either in it or thru the command line. Do you know which ones you changed and the command lines you finally used? I am at UChicago and we have a similar HPC setup so all else worked.

Thanks,
Sonakshi