Manual

General workflow of launchcontainers

By now, we assume that you have already git-cloned this repository at github/launchcontainers and installed all the required software.

There are two types of config files.

• The .yaml file that is used to control launchcontainers. This file will provide path information, which container to launch, in which HPC cluster, and with what parameters. We will call this lc_config file.
• The config files are specific to each container. They are required to control the analyses within the container. They are usually json files, but they can be yaml or other formats as well (we plan to migrate some of the json files to yaml in the future. IMPORTANT: some containers might have MORE THAN ONE CONFIG FILES. For example, rtp-pipeline, which has config.json and tractparams.csv. We will call this container_config file(s).

There are two main modes of working for launchcontainers:

• Prepare mode: Prepare the folders and files and container_config file(s), it will copy (or create symlinks) of the files, and then we will need to check that it is created correctly, and most importantly, edit the container_config file(s) so that the analysis will do what we want.
• Run mode: it will run the analysis by creating a job in the cluster, and running the container. It will check again that all the required files for this specific analysis/container are in place.

Main steps

These are the main steps to configure launchcontainers and run analyses within containers:

1. Create a project folder in your base directory:

   $ mkdir /TESTDATA/PROJECT_NAME and put your dicoms in the dicom/ directory

2. Convert your dicoms with heudiconv (heudiconv right now is not part of launchcontainers, remove this when added), and use NORDIC or presurfer or whatever you need to use to have the files in nifti format. RESULT: you should have a PROJECT_NAME/nifti/ folder, with all subjects and sessions there with all the nifti files organized in BIDS format

3. Copy launchcontainers/example_config.yaml (and launchcontainers/example_subSesList.txt) to /nifti folder (or any other place, but do not ever edit the one in /example_configs in the repo) and edit it with your paths and containers versions and HPC requirements. Follow the instructions on the yaml file to know what every field means. How to edit the yaml file should be auto-explained in the file itself. IMPORTANT: This file will be dynamic, every time we use it, it will be copied to the corresponding container/analysis folder, so that we have a copy of what was run in this specific analysis. You can edit this file as much as you need and delete it at the end. For example, if we only want to use one container, but we are testing some options in the configuration, we will keep editing this file and the analysis field will be changing, i.e "01", "02", "03", ... and every analysis-01 or analysis-02 folder will keep a copy of the right files. subSesList.txt only has a list with the subject id and the sessions in our project, and it contains a column to specify if we want to run this sub/ses or not.

   $ cp path_to_launchcontainers/example_configs/example_lc_config.yaml PROJECT_NAME/nifti/lc_config.yaml

   $ cp path_to_launchcontainers/example_configs/example_subSesList.txt PROJECT_NAME/nifti/subSesList.txt

4. Run launchcontainers in prepare mode.

   • (optional) Check the documentation and help for launchcontainer, it will give you some explanation on the flags:

     python path_to_launchcontainers/launchcontainers/launchcontainers.py -h

   • Use the prepare mode of launchcontainer to create symlinks and auto-check if all the required files are in place.

     python path_to_launchcontainers/launchcontainers/launchcontainers.py -lcc path_to_nifti_folder/lc_config.yaml -ssl path_to_nifti_folder/subSesList.txt -cc path_to_container_configs/config.json

   • The first time you run it, you can copy container_config files directly from the repo, so you can make path_to_container_configs = path_to_launchcontainers/example_configs/container_specific_configs/container/version/config.json, but after you edit your required parameters in the config file, here you could just use the newly edited config file, so for example this file will be in path_to_nifti/derivatives/container_name_version/analysis-analysis_name/config.json so that you could use this one and just edit/maintain the required analysis parameters.
   • If your container has more than one container_config, put them here one after the other separated with one white space .

5. Check that all the symlinks are successfully created and that the configs are copied

   • Use 'tree' command with color coding to check that there are no broken links)
   • 'cd' to path_to_nifti/derivatives/container_name_version/analysis-analysis_name/
   • Edit the container_config file(s) with the desired parameters. Please refer to each container's manual to know how to edit those config.json files. we will try to use yaml files so that the config files are auto-explanatory.

6. Run launchcontainers in run mode.

   • Type:

     $ python path_to_launchcontainers/launchcontainers/launchcontainers.py -lcc path_to_nifti/derivatives/container_name_version/analysis-analysis_name/config_lc.yaml -ssl path_to_nifti/derivatives/container_name_version/analysis-analysis_name/subSesList.txt --run_lc

     • If you do not edit the config_lc or subSesList.txt file in the path_to_nifti/derivatives/container_name_version/analysis-analysis_name folder, then it will be the same that it is in the path_to_nifti folder, so in this case you can reuse the previous command line. Nevertheless, we recommend the users take the habit of always specifying where is the lc_config file that we want to use, to reduce human errors.