Merge pull request #20 from flywheel-apps/add-configs

Update to 20.2.6, add --fs-subjects-dir, don't delete old runs

Dockerfile

@@ -1,4 +1,4 @@
-FROM nipreps/fmriprep:20.2.4
+FROM nipreps/fmriprep:20.2.6
 
 LABEL maintainer="[email protected]"
 

README.md

@@ -1,5 +1,5 @@
 # bids-fmriprep
-[Flywheel Gear](https://github.com/flywheel-io/gears/tree/master/spec) which runs [fMRIPrep](http://fmriprep.readthedocs.io) Long-Term Support version 20.2.4 (October 4, 2021) on BIDS-curated data. fMRIPrep is a functional magnetic resonance imaging (fMRI) data preprocessing pipeline that is designed to provide an easily accessible, state-of-the-art interface that is robust to variations in scan acquisition protocols and that requires minimal user input, while providing easily interpretable and comprehensive error and output reporting. It performs basic processing steps (coregistration, normalization, unwarping, noise component extraction, segmentation, skull stripping, etc.) providing outputs that can be easily submitted to a variety of group level analyses, including task-based or resting-state fMRI, graph theory measures, surface or volume-based statistics, etc.
+[Flywheel Gear](https://github.com/flywheel-io/gears/tree/master/spec) which runs [fMRIPrep](http://fmriprep.readthedocs.io) Long-Term Support version 20.2.6 (November 12, 2021) on BIDS-curated data. fMRIPrep is a functional magnetic resonance imaging (fMRI) data preprocessing pipeline that is designed to provide an easily accessible, state-of-the-art interface that is robust to variations in scan acquisition protocols and that requires minimal user input, while providing easily interpretable and comprehensive error and output reporting. It performs basic processing steps (coregistration, normalization, unwarping, noise component extraction, segmentation, skull stripping, etc.) providing outputs that can be easily submitted to a variety of group level analyses, including task-based or resting-state fMRI, graph theory measures, surface or volume-based statistics, etc.
 
 The version number is (Flywheel gear) MAJOR . MINOR . PATCH _ (algorithm) YY . MINOR . PATCH
 
@@ -8,7 +8,7 @@ This gear can only be run on datasets that have been BIDS curated and can pass t
 
 This Gear requires a (free) Freesurfer license. The license can be provided to the Gear in 3 ways. See [How to include a Freesurfer license file](https://docs.flywheel.io/hc/en-us/articles/360013235453).
 
-The bids-fmriprep Gear can run at the project, subject or session level.  Because files are in the BIDS format, all of the proper files will be used for the given session, subject, or separately, by subject, for the whole project.
+The bids-fmriprep Gear can run at the project, subject or session level.  Because files are in the BIDS format, all the proper files will be used for the given session, subject, or separately, by subject, for the whole project.
 
 ## Setup:
 Before running BIDS curation on your data, you must first prepare your data with the following steps:
@@ -33,7 +33,7 @@ Note that bids-fmriprep can take a *long* time to run because it runs Freesurfer
 
 ## Inputs
 
-Because the project has been BIDS curated, all of the proper T1, T2, and fMRI files will be automatically found.
+Because the project has been BIDS curated, all the proper T1, T2, and fMRI files will be automatically found.
 
 ### bidsignore (optional)
 A list of patterns (like .gitignore syntax) defining files that should be ignored by the
@@ -44,6 +44,9 @@ Your FreeSurfer license file. [Obtaining a license is free](https://surfer.nmr.m
 This file will be copied into the $FSHOME directory.  There are [three ways](https://docs.flywheel.io/hc/en-us/articles/360013235453-How-to-include-a-Freesurfer-license-file-in-order-to-run-the-fMRIPrep-gear-)
 to provide the license to this gear.  A license is required for this gear to run.
 
+## fs-subjects-dir
+Zip file of existing FreeSurfer subject's directory to reuse.  If the output of FreeSurfer recon-all is provided to fMRIPrep, that output will be used rather than re-running recon-all.  Unzipping the file should produce a particular subject's directory which will be placed in the $FREESURFER_HOME/subjects directory.  The name of the directory must match the -subjid as passed to recon-all.  This version of fMRIPrep uses Freesurfer v6.0.1.
+
 ## Config:
 Most config options are identical to those used in fmriprep, and so documentation can be found here https://fmriprep.readthedocs.io/en/stable/usage.html.
 

manifest.json

@@ -303,13 +303,13 @@
     }
   },
   "custom": {
-    "docker-image": "flywheel/bids-fmriprep:1.2.0_20.2.4",
+    "docker-image": "flywheel/bids-fmriprep:1.2.1_20.2.6",
     "flywheel": {
       "suite": "BIDS Apps"
     },
     "gear-builder": {
       "category": "analysis",
-      "image": "flywheel/bids-fmriprep:1.2.0_20.2.4"
+      "image": "flywheel/bids-fmriprep:1.2.1_20.2.6"
     },
     "license": {
       "dependencies": [
@@ -341,7 +341,7 @@
       "non-commercial-use-only": true
     }
   },
-  "description": "fMRIPrep 20.2.4 (Long-Term Support version) is a functional magnetic resonance imaging (fMRI) data preprocessing pipeline that is designed to provide an easily accessible, state-of-the-art interface that is robust to variations in scan acquisition protocols and that requires minimal user input, while providing easily interpretable and comprehensive error and output reporting. It performs basic processing steps (coregistration, normalization, unwarping, noise component extraction, segmentation, skullstripping etc.) providing outputs that can be easily submitted to a variety of group level analyses, including task-based or resting-state fMRI, graph theory measures, surface or volume-based statistics, etc.",
+  "description": "fMRIPrep 20.2.6 (Long-Term Support version) is a functional magnetic resonance imaging (fMRI) data preprocessing pipeline that is designed to provide an easily accessible, state-of-the-art interface that is robust to variations in scan acquisition protocols and that requires minimal user input, while providing easily interpretable and comprehensive error and output reporting. It performs basic processing steps (coregistration, normalization, unwarping, noise component extraction, segmentation, skullstripping etc.) providing outputs that can be easily submitted to a variety of group level analyses, including task-based or resting-state fMRI, graph theory measures, surface or volume-based statistics, etc.",
   "environment": {
     "PATH": "/usr/local/miniconda/bin:/opt/ICA-AROMA:/usr/lib/ants:/usr/lib/fsl/5.0:/usr/lib/afni/bin:/opt/freesurfer/bin:/bin:/opt/freesurfer/tktools:/opt/freesurfer/mni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
     "REQUESTS_CA_BUNDLE": "/etc/ssl/certs/ca-certificates.crt"
@@ -357,6 +357,11 @@
       "description": "FreeSurfer license file, provided during registration with FreeSurfer. This file will by copied to the $FSHOME directory and used during execution of the Gear.",
       "optional": true
     },
+    "fs-subjects-dir": {
+      "base": "file",
+      "description": "Zip file of existing FreeSurfer subject's directory to reuse.  If the output of FreeSurfer recon-all is provided to fMRIPrep, that output will be used rather than re-running recon-all.  Unzipping the file should produce a particular subject's directory which will be placed in the $FREESURFER_HOME/subjects directory.  The name of the directory must match the -subjid as passed to recon-all.  This version of fMRIPrep uses Freesurfer v6.0.1.",
+      "optional": true
+    },
     "key": {
       "base": "api-key",
       "read-only": true
@@ -368,5 +373,5 @@
   "name": "bids-fmriprep",
   "source": "https://github.com/nipreps/fmriprep",
   "url": "https://github.com/flywheel-apps/bids-fmriprep/blob/master/README.md",
-  "version": "1.2.0_20.2.4"
+  "version": "1.2.1_20.2.6"
 }

run.py

@@ -13,7 +13,7 @@
     build_command_list,
     exec_command,
 )
-from flywheel_gear_toolkit.utils.zip_tools import zip_output
+from flywheel_gear_toolkit.utils.zip_tools import unzip_archive, zip_output
 
 from utils.bids.download_run_level import download_bids_for_runlevel
 from utils.bids.run_level import get_analysis_run_level_and_hierarchy
@@ -172,6 +172,15 @@ def main(gtk_context):
         (subjects_dir / "fsaverage5").symlink_to(orig_subject_dir / "fsaverage5")
         (subjects_dir / "fsaverage6").symlink_to(orig_subject_dir / "fsaverage6")
 
+    subject_zip_file_path = gtk_context.get_input_path("fs-subjects-dir")
+    if subject_zip_file_path:
+        paths = list(Path("input/fs-subjects-dir").glob("*"))
+        log.info("Using provided Freesurfer subject file %s", str(paths[0]))
+        unzip_archive(paths[0], subjects_dir)
+
+        # Add --fs-subjects-dir argument to the command
+        config["fs-subjects-dir"] = subjects_dir
+
     environ["FS_LICENSE"] = str(FWV0 / "freesurfer/license.txt")
 
     license_list = list(Path("input/freesurfer_license").glob("*"))
@@ -252,11 +261,7 @@ def main(gtk_context):
 
             # This is what it is all about
             exec_command(
-                command,
-                environ=environ,
-                dry_run=dry_run,
-                shell=True,
-                cont_output=True,
+                command, environ=environ, dry_run=dry_run, shell=True, cont_output=True,
             )
 
     except RuntimeError as exc:

tests/conftest.py

@@ -40,7 +40,7 @@ def _method(zip_name):
         print(f'\ninstalling new gear, "{zip_name}"...')
         unzip_archive(gear_tests / zip_name, str(FWV0))
 
-        # The "freesurfer" direcory needs to have the standard freesurfer
+        # The "freesurfer" directory needs to have the standard freesurfer
         # "subjects" directory and "license.txt" file.
 
     return _method

tests/integration_tests/test_dry_run.py

@@ -30,6 +30,7 @@ def test_dry_run_works(
 
         assert status == 0
         assert (FWV0 / "work/bids/.bidsignore").exists()
+        assert (FWV0 / "freesurfer/subjects/sub-42/label/empty").exists()
         assert search_caplog_contains(caplog, "command is", "participant")
         assert search_caplog_contains(caplog, "command is", "'arg1', 'arg2'")
         assert search_caplog(caplog, "No BIDS errors detected.")

tests/integration_tests/test_wet_run_errors.py

@@ -17,6 +17,9 @@ def test_wet_run_errors(
     if not user_json.exists():
         TestCase.skipTest("", f"No API key available in {str(user_json)}")
 
+    # This fake gear must have a destination that has an analysis on a session that has
+    # no bold scans (like BIDS_multi_session/ses-Session2).  It downloads the BIDS data
+    # for that session and the lack of a bold scan will cause the expected error.
     install_gear("wet_run.zip")
 
     with flywheel_gear_toolkit.GearToolkitContext(input_args=[]) as gtk_context:

utils/singularity.py

@@ -43,12 +43,9 @@ def run_in_tmp_dir():
     else:
         log.debug("Running in %s", running_in)
 
-    # remove any previous runs (possibly left over from previous testing)
-    previous_runs = list(Path("/tmp").glob(f"{SCRATCH_NAME}*"))
-    log.debug("previous_runs = %s", previous_runs)
-    for prev in previous_runs:
-        log.debug("rm %s", prev)
-        shutil.rmtree(prev)
+    # This used to remove any previous runs (possibly left over from previous testing) but that would be bad
+    # if other bids-fmripreps are running on shared hardware at the same time because their directories would
+    # be deleted mid-run.  A very confusing error to debug!
 
     # Create temporary place to run gear
     WD = tempfile.mkdtemp(prefix=SCRATCH_NAME, dir="/tmp")