Add microscopy tools - CryoEM -relion (#612)

* relion container (without ctffind, motioncor2, topaz)

* added relion environment variables and topaz container

* updates to README.md

recipes/relion/README.md

@@ -0,0 +1,29 @@
+
+----------------------------------
+## relion/toolVersion ##
+RELION (for REgularised LIkelihood OptimisatioN) is a stand-alone computer program for Maximum A Posteriori (MAP) refinement of (multiple) 3D reconstructions or 2D class averages in cryo electron microscopy (cryo-EM).
+
+Example:
+```
+```
+
+More documentation can be found here: https://github.com/3dem/relion/tree/toolVersion
+
+To make the executables and scripts inside this container transparently available in the command line of environments where Neurocommand is installed (without the need to use 'Apptainer exec'): ml relion/toolVersion
+
+The packages included in this container, their version, and the base folder of the installation within the container:
+
+relion - toolVersion - /opt/relion-toolVersion
+
+ctffind - 4.1.14 - /opt/ctffind-4.1.14
+
+motioncor2 - 1.6.4 - /opt/motioncor2-1.6.4
+
+cudatoolkit - 11.8 - /usr/local/cuda-11.8
+
+Citation:
+```
+
+```
+
+----------------------------------

recipes/relion/app_test.sh

@@ -0,0 +1,91 @@
+#
+# app_test.sh - Application testing script
+# ========================================
+#
+# This script should contain an automatic functional test of the application that makes it possible to test if everything was correctly installed within the container.
+# The Neurodesktop administrator will run it before every release of Neurodesktop.
+# If the test fails, the administrator will try to fix the problem (possible contacting you about it), or alternatively, mark the container as faulty in the specific Neurodesktop release.
+#
+# The script:
+# - should return 0 exit code if the container passed the test
+# - should return a non-zero exit code if the container failed the test, and an error message (if available) should be printed to stderr 
+#   (to send output of any bash command to stderr, add "1>&2" at the end of the command)
+# - should return 1 exit code and print "N/A" to stderr if the container developer hasn't written a test script yet
+#   (that will be the default if the developer uses this example as-is)
+# Notide that any other stderr or stdout will be captured when running the script and provided to the administrator as part of the testing. That said,
+# given that many applications are tested, the administrator may not monitor the output and rely primarily on the exit code.
+#
+# The script must be a bash script, as it will be executed using 'bash -e /neurodesk/app_test.sh'. 
+# The '-e' flag indicates to bash that if there is an error in the script, it should quit immediately with the non-zero exit code of the error rather than the bash default of continuing script execution after an error. This way errors in the script won't be overlooked during testing (the script continuing and returning zero exit code on termination, the administrator might not notice the error message).
+#
+# Notice that the script can also be executed by users directly by running /neurodesk/test.sh within the container. This can be used to double-check that the application works properly in the specific environment used by the user (although Singularity containers are supposed to run identically regardless of the execution environment, there are some exceptions).
+#
+
+############################################################
+# Uncomment the line below when you test script is complete
+############################################################
+echo 'N/A' 1>&2; exit 1
+
+
+####################################################################################
+# The commands below provide an example for a test script. Please edit as necessary
+# When done, remove the line above and create the container
+#
+# After the container is incorporated into Neurocontainers and being built by the CI, 
+# convert it into a local sif file using the /neurocommand/local/fetch_and_run.sh command provided 
+# in the "New container ..." issue confirming the container was built.
+#
+# If you see that the container runs successfully as a sif file, run the following commands to verify your test scripts return the appropriate output
+# when being called using a singularity exec command (using your package NAME, VERSION, and BUILDDATE):
+#
+# singularity --silent exec --pwd /tmp /neurocommand/local/containers/NAME_VERSION_BUILDDATE/NAME_VERSION_BUILDDATE.simg /bin/bash -e /neurodesk/app_test.sh 1>stdout 2>stderr
+# echo 'EXIT CODE: '$?
+# echo 'STDOUT:'
+# cat stdout
+# echo 'STDERR:'
+# cat stderr
+#
+####################################################################################
+
+# The variables below should be set according to the tested app. They are just used as an example
+URL='https://download_url'               # URL of test data
+EXEC='process'                           # executable of app
+ARGUMENTS='-all -inv input'         	 # arguments for executable
+
+# download test data (in this case, we assume it includes a folder 'input' with the input data, and a folder 'output' with expected output data.
+if curl -L "$URL"  > download.zip && unzip download.zip
+then
+	   echo 'app_test.sh: Downloaded data from '"$URL"' successfuly'
+else
+	   exit_code=$?
+	   echo 'app_test.sh: Cannot download test data from '"$URL"'. Return non-zero exit code' 1>&2
+	   exit "$exit_code"
+fi
+
+# if exists, delete test output folder, to make sure we do not use output of previous tests
+if [ -d test_output ]
+then
+	    rm -Rf test_output
+fi
+
+# execute app
+if "$EXEC" ${ARGUMENTS} test_output
+then
+	   echo 'app_test.sh: found '"$EXEC"' and executed it. It returned 0 exit code'
+else
+ 	  exit_code=$?
+	   echo 'app_test.sh: executing '"$EXEC"' returned an error. Return non-zero exit code' 1>&2
+	   exit "$exit_code"
+fi
+
+# compare test output folder with expected output folder
+if diff -r test_output output
+then
+	echo 'app_test.sh: test successful'
+else
+	exit_code=$?
+	echo 'app_test.sh: Generated output does not match expected output. Return non-zero exit code' 1>&2
+	exit "$exit_code"
+fi
+
+

recipes/relion/app_version.sh

@@ -0,0 +1,59 @@
+
+# app_version.txt - print the version of the application
+# ======================================================
+#
+# this script should print out the version of the software in order to verify that the correct version is installed
+# please retrieve the version by running one of the executables with a correct flag rather than simply hard-coding it. This ensures that the executable is of the correct version
+# The Neurodesktop administrator will run it before every release of Neurodesktop.
+# If the test return a version number that diverges from the expected one, the administrator will try to fix the problem (possible contacting you about it), or alternatively, mark the container as faulty in the specific Neurodesktop release.
+#
+# The script:
+# - should return exit code 0 and print the version of the package (in exactly the same format used in Neurodesk) to stdout
+# - should return 1 exit code and print "N/A" to stderr if the container developer hasn't included this functionality yet (that will be the default if the script provided in the template has not been modified)
+# - should return non-zero exit code if the version cannot be retrieved, and an error message (if available) should be printed to stderr
+#
+# Notide that any other stderr or stdout will be captured when running the script and provided to the administrator as part of the testing. That said,
+# given that many applications are tested, the administrator may not monitor the output and rely primarily on the exit code.
+#
+# The script must be a bash script, as it will be executed using 'bash -e /neurodesk/app_version.sh'. 
+# The '-e' flag indicates to bash that if there is an error in the script, it should quit immediately with the non-zero exit code of the error rather than the bash default of continuing script execution after an error. This way errors in the script won't be overlooked during testing (the script continuing and returning zero exit code on termination, the administrator might not notice the error message).
+#
+# Notice that the script can also be executed by users directly by running /neurodesk/app_version.sh within the container. This can be used to double-check that the application works properly in the specific environment used by the user (although Singularity containers are supposed to run identically regardless of the execution environment, there are some exceptions).
+#
+
+####################################################################
+# Uncomment the line below when your version test script is complete
+#####################################################################
+echo 'N/A' 1>&2; exit 1
+
+####################################################################################
+# The commands below provide an example for a version printing script. Please edit as necessary
+#
+# After the container is incorporated into Neurocontainers and being built by the CI, 
+# convert it into a local sif file using the /neurocommand/local/fetch_and_run.sh command provided 
+# in the "New container ..." issue confirming the container was built.
+#
+# If you see that the container runs successfully as a sif file, run the following commands to verify your test scripts return the appropriate output
+# when being called using a singularity exec command (using your package NAME, VERSION, and BUILDDATE):
+# 
+# singularity --silent exec --pwd /tmp /neurocommand/local/containers/NAME_VERSION_BUILDDATE/NAME_VERSION_BUILDDATE.simg /bin/bash -e /neurodesk/app_version.sh 1>stdout 2>stderr
+# echo 'EXIT CODE: '$?
+# echo 'STDOUT:'
+# cat stdout
+# echo 'STDERR:'
+# cat stderr
+#
+####################################################################################
+
+EXEC='process'                           # executable of app
+ARGUMENTS='-v'		         	 # arguments for executable
+
+# an example for the version number being printed by the executable in the second line of output
+if "$EXEC" ${ARGUMENTS} | head -2 | tail -1
+then
+    a=1 # empty command, just so I'll have a then clause. Cannot run mfcsc without if, because then will exit without explaining what was the problem
+else
+    exit_code=$?
+    echo 'app_version.sh: executing '"$EXEC"' returned an error. Return non-zero exit code' 1>&2
+    exit "$exit_code"
+fi

recipes/relion/build.sh

@@ -0,0 +1,105 @@
+# this template file builds datalad and is then used as a docker base image for layer caching + it contains examples for various things like github install, curl, ...
+export toolName='relion'
+# toolName or toolVersion CANNOT contain capital letters or dashes or underscores (Docker registry does not accept this!)
+
+export toolVersion='4.0.1' 
+# the version number cannot contain a "-" - try to use x.x.x notation always
+# toolVersion will automatically be written into README.md - for this to work leave "toolVersion" in the README unaltered.
+
+export COMPUTE_CAPABILITY=$(nvidia-smi --query-gpu=compute_cap --format=csv | sed -ne '2 s/\.// p')
+# get the Compute Capability of the GPU to compile relion with the right GPU architecture
+
+export CTFFIND_VERSION='4.1.14'
+export CTFFIND_LINK='https://grigoriefflab.umassmed.edu/system/tdf?path=ctffind-4.1.14.tar.gz&file=1&type=node&id=26'
+# ctffind version and download link
+
+export MOTIONCOR2_VERSION='1.6.4'
+export MOTIONCOR2_LINK='https://drive.google.com/uc?export=download&id=1hskY_AbXVgrl_BUIjWokDNLZK0c1FLxF'
+# motioncor2 version and download link (the link may not work forever, and may need to be updated or changed to a local install)
+
+export GO_VERSION='1.17.2'
+export SINGULARITY_VERSION='3.9.3'
+export OS=linux 
+export ARCH=amd64
+# GO and singularity version to run modules with lmod
+
+# !!!!
+# You can test the container build locally by running `bash build.sh -ds`
+# !!!!
+
+if [ "$1" != "" ]; then
+    echo "Entering Debug mode"
+    export debug=$1
+fi
+source ../main_setup.sh
+###########################################################################################################################################
+# IF POSSIBLE, PLEASE DOCUMENT EACH ARGUMENT PROVIDED TO NEURODOCKER. USE THE `# your comment` NOTATION THAT ALLOWS MID-COMMAND COMMENTS
+# NOTE 1: THE QUOTES THAT ENCLOSE EACH COMMENT MUST BE BACKQUOTES (`). OTHER QUOTES WON'T WORK!
+# NOTE 2: THE BACKSLASH (\) AT THE END OF EACH LINE MUST FOLLOW THE COMMENT. A BACKSLASH BEFORE THE COMMENT WON'T WORK!
+# NOTE 3: COMMENT LINES, I.E. LINES THAT START WITH #, CANNOT BE INCLUDED IN THE MIDDLE OF THE neurodocker generate COMMAND. INSTEAD,
+#         USE AN EMPTY LINE AND PUT YOUR COMMENT AT THE END USING THIS FORMAT: `# your comment goes here` \ 
+##########################################################################################################################################
+neurodocker generate ${neurodocker_buildMode} \
+   --base-image ubuntu:22.04                		`# RELION: Using Ubuntu 22.04 because not all required packages were available on neurodebian` \
+   --env DEBIAN_FRONTEND=noninteractive                 `# RECOMMENDED TO KEEP AS IS: this disables interactive questions during package installs` \
+   --pkg-manager apt                                    `# RECOMMENDED TO KEEP AS IS: desired package manager, has to match the base image (e.g. debian needs apt; centos needs yum)` \
+   --run="printf '#!/bin/bash\nls -la' > /usr/bin/ll"   `# RECOMMENDED TO KEEP AS IS: define the ll command to show detailed list including hidden files`  \
+   --run="chmod +x /usr/bin/ll"                         `# RECOMMENDED TO KEEP AS IS: make ll command executable`  \
+   --run="mkdir -p ${mountPointList}"                   `# MANDATORY: create folders for singularity bind points` \
+   --install wget git curl ca-certificates unzip        `# RECOMMENDED: install system packages` \
+             cmake git build-essential mpi-default-bin mpi-default-dev libfftw3-dev libtiff-dev libpng-dev ghostscript libxft-dev	`# RELION: install relion dependencies` \
+             libwxgtk3.0-gtk3-dev			`# CTFFIND: install wx-config-3.0 for ctffind-4.1.14` \
+             lmod 					`# TOPAZ: install lmod to run modules, like topaz` \
+   --workdir=/tmp \
+   --run="wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin	`# RELION: steps to install CUDA 11.8 from https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local` \
+       && mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 \
+       && wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb \
+       && dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb \
+       && cp /var/cuda-repo-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/" \
+   --install cuda-toolkit-11-8				`# RELION: install CUDA Toolkit 11.8` \
+   --run="git clone 'https://github.com/3dem/relion.git' --branch=${toolVersion}	`# RELION: clone relion git repository` \
+       && cd relion && mkdir build && cd build		`# RELION: create and move into build directory` \
+       && cmake -DCUDA_ARCH=${COMPUTE_CAPABILITY} -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DCMAKE_INSTALL_PREFIX=/opt/${toolName}-${toolVersion}/ -DFORCE_OWN_FLTK=ON ..	`# RELION: Compile with GPU architecture and CUDA version` \
+       && make && make install" \
+   --run="wget -O ctffind-${CTFFIND_VERSION}.tar.gz '${CTFFIND_LINK}'	`# CTFFIND: download and install ctffind` \
+       && tar -xf ctffind-${CTFFIND_VERSION}.tar.gz \
+       && cd ctffind-${CTFFIND_VERSION} \
+       && ./configure --disable-debugmode --enable-mkl --prefix=/opt/ctffind-${CTFFIND_VERSION}/ \
+       && make && make install" \
+   --run="wget -O MotionCor2_${MOTIONCOR2_VERSION}.zip '${MOTIONCOR2_LINK}'	`# MOTIONCOR2: download and install motioncor2` \
+       && mkdir -p /opt/motioncor2-${MOTIONCOR2_VERSION}/bin/ \
+       && unzip MotionCor2_${MOTIONCOR2_VERSION}.zip -d /opt/motioncor2-${MOTIONCOR2_VERSION}/bin/" \
+   --copy load_topaz.sh /opt/${toolName}-${toolVersion}/        `# TOPAZ: copy script to launch topaz` \
+   --run="chmod +x /opt/${toolName}-${toolVersion}/load_topaz.sh" \
+   --env RELION_CTFFIND_EXECUTABLE=/opt/ctffind-${CTFFIND_VERSION}/bin/ctffind    `# CTFFIND: relion environment variable for ctffind` \
+   --env RELION_MOTIONCOR2_EXECUTABLE=/opt/motioncor2-${MOTIONCOR2_VERSION}/bin/MotionCor2_${MOTIONCOR2_VERSION}_Cuda118_Mar312023	`# MOTIONCOR2: relion environment variable for motioncor2` \
+   --env RELION_TOPAZ_EXECUTABLE=/opt/${toolName}-${toolVersion}/load_topaz	`# TOPAZ: relion environment variable for Topaz` \
+   --env PATH='$PATH':/opt/${toolName}-${toolVersion}/bin `# MANDATORY: add your tool executables to PATH` \
+   --env DEPLOY_PATH=/opt/${toolName}-${toolVersion}/bin/ `# MANDATORY: define which directory's binaries should be exposed to module system (alternative: DEPLOY_BINS -> only exposes binaries in the list)` \
+   --env GOPATH='$HOME'/go 				`# TOPAZ: install GO to compile singularity` \
+   --env PATH='$PATH':/usr/local/go/bin:'$PATH':${GOPATH}/bin \
+   --run="wget https://dl.google.com/go/go$GO_VERSION.$OS-$ARCH.tar.gz \
+       && tar -C /usr/local -xzvf go$GO_VERSION.$OS-$ARCH.tar.gz \
+       && rm go$GO_VERSION.$OS-$ARCH.tar.gz \
+       && mkdir -p $GOPATH/src/github.com/sylabs \
+       && cd $GOPATH/src/github.com/sylabs \
+       && wget https://github.com/sylabs/singularity/releases/download/v${SINGULARITY_VERSION}/singularity-ce-${SINGULARITY_VERSION}.tar.gz \
+       && tar -xzvf singularity-ce-${SINGULARITY_VERSION}.tar.gz \
+       && cd singularity-ce-${SINGULARITY_VERSION} \
+       && ./mconfig --without-suid --prefix=/usr/local/singularity \
+       && make -C builddir \
+       && make -C builddir install \
+       && cd .. \
+       && rm -rf singularity-ce-${SINGULARITY_VERSION} \
+       && rm -rf /usr/local/go $GOPATH \
+       && ln -s /usr/local/singularity/bin/singularity /bin/" \
+   --copy module.sh /usr/share/ 			`# TOPAZ: copy module file for lmod` \
+   --copy README.md /README.md                          `# MANDATORY: include readme file in container` \
+   --copy license.txt /license.txt                      `# MANDATORY: include license file in container` \
+   --copy * /neurodesk/                              	`# MANDATORY: copy test scripts to /neurodesk folder - build.sh will be included as well, which is a good idea` \
+   --run="chmod +x /neurodesk/*.sh"                     `# MANDATORY: allow execution of all shell scripts in /neurodesk inside the container` \
+  > ${imageName}.${neurodocker_buildExt}                `# THIS IS THE LAST COMMENT; NOT FOLLOWED BY BACKSLASH!`
+
+if [ "$1" != "" ]; then
+   ./../main_build.sh
+fi

recipes/relion/load_topaz.sh

@@ -0,0 +1,3 @@
+#!/bin/bash'
+module load topaz/latest
+topaz '$@'

recipes/relion/module.sh

@@ -0,0 +1,14 @@
+# system-wide profile.modules                                          #
+# Initialize modules for all sh-derivative shells                      #
+#----------------------------------------------------------------------#
+trap "" 1 2 3
+
+case "$0" in
+    -bash|bash|*/bash) . /usr/share/lmod/6.6/init/bash ;;
+       -ksh|ksh|*/ksh) . /usr/share/lmod/6.6/init/ksh ;;
+       -zsh|zsh|*/zsh) . /usr/share/lmod/6.6/init/zsh ;;
+          -sh|sh|*/sh) . /usr/share/lmod/6.6/init/sh ;;
+                    *) . /usr/share/lmod/6.6/init/sh ;;  # default for scripts
+esac
+
+trap - 1 2 3