README.md

Warning

A nf-scil legacy dependency in git-lfs, which is affected by recent git security updates, may pose problems. Cloning the repository may generate an error, even if the procedure was successful. To disable the behavior, until the problem is patched, prepend your git clone command with GIT_CLONE_PROTECTION_ACTIVE=false or add it to your environment.

Welcome to the nf-scil project ! A Nextflow modules and workflows repository for neuroimaging maintained by the SCIL team. The primary focus of the library is to provide pre-built processes and processing sequences for diffusion Magnetic Resonance Imaging, optimized for Nextflow DSL2, based on open-source technologies and made easily available to pipeline's developers through the nf-core framework.

• Using modules from nf-scil
• Developing in nf-scil
  • Manual configuration
    • Dependencies
    • Python environment
    • Loading the project's environment
    • Working with VS Code
  • Configuration via the devcontainer
  • Contributing to the nf-scil project
    • Adding a new module to nf-scil
      • Generate the template
      • Edit the template
        • Editing the module's main.nf
        • Editing the module's meta.yml
        • Editing the module's environment.yml
        • Editing the tests' main.nf.test
        • Editing the tests' nextflow.config
      • Run the tests to generate the test metadata file
      • Lint your code
      • Last safety test
      • Submit your PR
    • Defining processes optional parameters
    • Test data infrastructure
      • Using the .test_data directory
      • Using Scilpy Fetcher
    • Adding a new subworkflow to nf-scil
      • Generate the template
      • Generate the template
        • Edit the subworkflow's main.nf
          • Define your Subworkflow inputs.
          • Fill the main: section.
          • Define your Workflow outputs.
        • Edit the subworkflow's meta.yml
      • Lint your code
      • Submit your PR
  • Running tests
  • Configuring Docker for easy usage
  • Installing Prettier

Using modules from nf-scil

To import modules from nf-scil, you first need to install nf-core on your system (can be done simply using pip install nf-core). Once done, nf-scil modules are imported using this command :

nf-core modules \
  --git-remote https://github.com/scilus/nf-scil.git \
  install <category>/<tool>

where you input the <tool> you want to import from the desired <category>. To get a list of the available modules, run :

nf-core modules \
  --git-remote https://github.com/scilus/nf-scil.git \
  list remote

The same can be done for subworkflows, replacing modules in the nf-core command by `subworkflows, e.g. :

nf-core subworkflows \
  --git-remote https://github.com/scilus/nf-scil.git \
  install <category>/<subworkflow>

It can become heavy to always prepend the commands with --git-remote, even so if you need to specify a --branch where to fetch the information. You can instead define the git-remote and branch using Environment Variables :

export NFCORE_MODULES_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_MODULES_GIT_BRANCH=main
export NFCORE_SUBWORKFLOWS_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_SUBWORKFLOWS_GIT_BRANCH=main

and call all commands without specifying the --git-remote and --branch options, while still targeting the nf-scil repository.

Developing in nf-scil

The nf-scil project requires some specific tools to be installed on your system so that the development environment runs correctly. You can install them manually, but if you desire to streamline the process and start coding faster, we highly recommend using the VS Code development container to get fully configured in a matter of minutes.

Manual configuration

Dependencies

• Python ≥ 3.8, < 3.13
• Docker ≥ 24 (we recommend using Docker Desktop)
• Java Runtime ≥ 11, ≤ 17
  • On Ubuntu, install openjdk-jre-<version> packages
• Nextflow ≥ 21.04.3
• nf-test ≥ 0.9.0-rc1
• Node ≥ 14 and Prettier (see below)

Important

Nextflow might not detect the right Java virtual machine by default, more so if multiple versions of the runtime are installed. If so, you need to set the environment variable JAVA_HOME to target the right one.

• Linux : look in /usr/lib/jvm for a folder named java-<version>-openjdk-<platform> and use it as JAVA_HOME.
• MacOS : if the Java jvm is the preferential one, use JAVA_HOME=$(/usr/libexec/java_home). Else, look into /Library/Java/JavaVirtualMachines for the folder with the correct runtime version (named jdk<inner version>_1<runtime version>.jdk) and use the following : /Library/Java/JavaVirtualMachines/dk<inner version>_1<runtime version>.jdk/Contents/Home.

Python environment

The project uses poetry to manage python dependencies. To install it using pipx, run the following commands :

pip install pipx
pipx ensurepath
pipx install poetry

Note

If the second command above fails, pipx cannot be found in the path. Prepend the second command with $(which python) -m and rerun the whole block.

Warning

Poetry doesn't like when other python environments are activated around it. Make sure to deactivate any before calling poetry commands.

Once done, install the project with :

poetry install

Loading the project's environment

Important

Make sure no python environment is activated before running commands !

The project scripts and dependencies can be accessed using :

poetry shell

which will activate the project's python environment in the current shell.

Note

You will know the poetry environment is activated by looking at your shell. The input line should be prefixed by : (nf-scil-tools-py<version>), with <version> being the actual Python version used in the environment.

To exit the environment, simply enter the exit command in the shell.

Important

Do not use traditional deactivation (calling deactivate), since it does not relinquish the environment gracefully, making it so you won't be able to reactivate it without exiting the shell.

Global environment

Set the following environment variables in your .bashrc (or whatever is the equivalent for your shell) :

export NFCORE_MODULES_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_MODULES_GIT_BRANCH=main
export NFCORE_SUBWORKFLOWS_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_SUBWORKFLOWS_GIT_BRANCH=main

This will make it so the nf-core commands target the right repository by default. Else, you'll need to add --git-remote and --branch options to pretty much all commands relating to modules and subworkflows.

Working with VS Code

The nf-scil project curates a bundle of useful extensions for Visual Studio Code, the nf-scil-extensions package. You can find it easily on the extension marketplace.

Configuration via the devcontainer :

The devcontainer definition for the project contains all required dependencies and setup steps are automatically executed. To use this installation method, you need to have Docker (refer to this section for configuration requirements or validate your configuration) and Visual Studio Code installed on your system.

Open the cloned repository in VS Code and click on the arrow box in the lower left corner, to get a prompt to Reopen in container. The procedure will start a docker build, wait for a few minutes and enjoy your fully configured development environment.

• Available in the container :

  • nf-scil, nf-core all accessible through the terminal, which is configured to load the poetry environment in shells automatically
  • nf-scil configured as the main repository for all nf-core commands, using NFCORE_* environment variables
  • git, git-lfs, github-cli
  • curl, wget, apt-get
  • nextflow, nf-test, docker, tmux

• Available in the VS Code IDE through extensions :

  • Docker images and containers management
  • Python and C++ linting, building and debugging tools
  • Github Pull Requests management
  • Github flavored markdown previewing

Contributing to the nf-scil project

If you want to propose a new module to the repository, follow the guidelines in the module creation documentation. The same goes for subworkflows, using these guidelines instead. We follow standards closely aligned with nf-core, with some exceptions on process atomicity and how test data is handled. Modules that don't abide to them won't be accepted and PR containing them will be closed automatically.

Running tests

Tests are run through nf-core, using the command :

nf-core modules test <category/tool>

The tool can be omitted to run tests for all modules in a category.

Configuring Docker for easy usage

The simplest way of installing everything Docker is to use Docker Desktop. You can also go the engine way and install Docker manually.

Once installed, you need to configure some access rights to the Docker daemon. The easiest way to do this is to add your user to the docker group. This can be done with the following command :

sudo groupadd docker
sudo usermod -aG docker $USER

After running this command, you need to log out and log back in to apply the changes.

Installing Prettier

To install Prettier for the project, you need to have node and npm installed on your system to at least version 14. On Ubuntu, you can do it using snap :

sudo snap install node --classic

However, if you cannot install snap, or have another OS, refer to the official documentation for the installation procedure.

Under the current configuration for the Development Container, for this project, we use the following procedure, considering ${NODE_MAJOR} is at least 14 for Prettier :

curl -fsSL https://deb.nodesource.com/setup_${NODE_MAJOR}.x | bash - &&\
apt-get install -y nodejs

npm install --save-dev --save-exact prettier

echo "function prettier() { npm exec prettier $@; }" >> ~/.bashrc