ENH: Restore carpetplot and other final adjustments (#3131)

## Changes proposed in this pull request

Closing in on the alpha. Getting tests running, but hopefully we'll
finish up the reports and cleanup tomorrow.

docs/installation.rst

@@ -6,7 +6,7 @@ Installation
 There are two ways to install *fMRIPrep*:
 
 * using container technologies (RECOMMENDED); or
-* within a `Manually Prepared Environment (Python 3.8+)`_, also known as
+* within a `Manually Prepared Environment (Python 3.10+)`_, also known as
   *bare-metal installation*.
 
 The ``fmriprep`` command-line adheres to the `BIDS-Apps recommendations
@@ -42,8 +42,18 @@ or `Singularity <https://www.nipreps.org/apps/singularity/>`__ subsections.
 The *NiPreps* portal also contains
 `extended details of execution with the Docker wrapper <https://www.nipreps.org/apps/docker/#running-a-niprep-with-a-lightweight-wrapper>`__.
 
-Manually Prepared Environment (Python 3.8+)
-===========================================
+In short, install the ``fmriprep-docker`` wrapper with pip::
+
+  $ python -m pip install fmriprep-docker
+
+Then run the ``fmriprep-docker`` command-line as if you were running
+``fmriprep`` on a *bare-metal* installation::
+
+  $ fmriprep-docker <input_bids_path> <derivatives_path> <analysis_level> <named_options>
+
+
+Manually Prepared Environment (Python 3.10+)
+============================================
 
 .. warning::
 
@@ -56,7 +66,7 @@ A relatively interpretable description of how your environment can be set-up
 is found in the `Dockerfile <https://github.com/nipreps/fmriprep/blob/master/Dockerfile>`_.
 As an additional installation setting, FreeSurfer requires a license file (see :ref:`fs_license`).
 
-On a functional Python 3.8 (or above) environment with ``pip`` installed,
+On a functional Python 3.10 (or above) environment with ``pip`` installed,
 *fMRIPrep* can be installed using the habitual command ::
 
     $ python -m pip install fmriprep

docs/outputs.rst

@@ -62,6 +62,28 @@ This layout, now the default, may be explicitly specified with the
 For compatibility with versions of fMRIPrep prior to 21.0, the
 `legacy layout`_ is available via ``--output-layout legacy``.
 
+Processing level
+----------------
+As of version 23.2.0, fMRIPrep supports three levels of derivatives:
+
+* ``--level minimal``: This processing mode aims to produce the smallest
+  working directory and output dataset possible, while enabling all further
+  processing results to be deterministically generated. Most components of
+  the `visual reports`_ can be generated at this level, so the quality of
+  preprocessing can be assessed. Because no resampling is done, confounds
+  and carpetplots will be missing from the reports.
+* ``--level resampling``: This processing mode aims to produce additional
+  derivatives that enable third-party resampling, resampling BOLD series
+  in the working directory as needed, but these are not saved to the output
+  directory.
+  The ``--me-output-echos`` flag will be enabled at this level, in which
+  case the individual echos will be saved to the working directory after
+  slice-timing correction, head-motion correction, and susceptibility
+  distortion correction.
+* ``--level full``: This processing mode aims to produce all derivatives
+  that have previously been a part of the fMRIPrep output dataset.
+  This is the default processing level.
+
 Visual Reports
 --------------
 *fMRIPrep* outputs summary reports, written to ``<output dir>/fmriprep/sub-<subject_label>.html``.

docs/usage.rst

@@ -128,10 +128,13 @@ would be equivalent to the latest example: ::
 
 Reusing precomputed derivatives
 -------------------------------
+
 Reusing a previous, partial execution of *fMRIPrep*
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 *fMRIPrep* will pick up where it left off a previous execution, so long as the work directory
 points to the same location, and this directory has not been changed/manipulated.
+Some workflow nodes will rerun unconditionally, so there will always be some amount of
+reprocessing.
 
 Using a previous run of *FreeSurfer*
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -143,48 +146,39 @@ You can use the ``--fs-subjects-dir`` flag to specify a different location to sa
 FreeSurfer outputs.
 If precomputed results are found, they will be reused.
 
-The *anatomical fast-track*
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Starting with version 20.1.0, *fMRIPrep* has a command-line argument (``--anat-derivatives <PATH>``)
-to indicate a path from which the preprocessed information derived from the T1w, T2w (if present) and
-FLAIR (if present) images.
-This feature was envisioned to help process very large multi-session datasets where the anatomical
-images can be averaged (i.e., anatomy is not expected to vary substantially across sessions).
-An example where this kind of processing would be useful is
-`My Connectome <https://openneuro.org/datasets/ds000031/>`__, a dataset that contains
-107 sessions for a single-subject.
-Most of these sessions contain anatomical information which, given the design of the dataset, can be averaged
-across sessions as no substantial changes should happen.
-In other words, the anatomical information of the dataset can be considered as *cross-sectional*.
-Before version 20.1.0, preprocessing this dataset would be hard for two limitations:
-
-  * if the dataset were to be processed in just one enormous job (be it in a commercial Cloud or
-    :abbr:`HPC (high-performance computing)` resources), the amount of data to be processed surely
-    would exceed the time limitations per job (and/or related issues, such as restarting from where
-    it left before); or
-  * if the processing were `split in sessions <https://github.com/nipreps/fmriprep/issues/1175>`__,
-    then *fMRIPrep* would attempt to re-process the anatomical information for every session.
-
-Because processing this emerging type of datasets (*densely sampled neuroimaging*) was impractical with
-*fMRIPrep*, the option ``--anat-derivatives`` will shortcut the whole anatomical processing.
-
-.. danger::
-    Using the *anatomical fast-track* (the ``--anat-derivatives`` argument) has important side-effects
-    that risk the reproducibility and reliability of *fMRIPrep*.
-    This flag breaks *fMRIPrep*'s internal tracing of provenance, and it trusts whatever input *fMRIPrep*
-    is given (so long it is BIDS-Derivatives compliant and contains all the necessary files).
-
-    When reporting results obtained with ``--anat-derivatives``, please make sure you highlight this
-    particular deviation from *fMRIPrep*, and clearly describe the alternative preprocessing of
-    anatomical data.
-
-.. attention::
-    When the intention is to combine the *anatomical fast-track* with some advanced options that involve
-    standard spaces (e.g., ``--cifti-output``), please make sure you include the
-    ``MNI152NLin6Asym`` space to the ``--output-spaces`` list in the first invocation of *fMRIPrep*
-    (or *sMRIPrep*) from which the results are to be reused.
-    Otherwise, a warning message indicating that *fMRIPrep*'s expectations were not met will be issued,
-    and the pre-computed anatomical derivatives will not be reused.
+BIDS Derivatives reuse
+~~~~~~~~~~~~~~~~~~~~~~
+As of version 23.2.0, *fMRIPrep* can reuse precomputed derivatives that follow BIDS Derivatives
+conventions. To provide derivatives to *fMRIPrep*, use the ``--derivatives`` (``-d``) flag one
+or more times.
+
+This mechanism replaces the earlier, more limited ``--anat-derivatives`` flag.
+
+.. note::
+   Derivatives reuse is considered *experimental*.
+
+This feature has several intended use-cases:
+
+  * To enable fMRIPrep to be run in a "minimal" mode, where only the most essential
+    derivatives are generated. This can be useful for large datasets where disk space
+    is a concern, or for users who only need a subset of the derivatives. Further
+    derivatives may be generated later, or by a different tool.
+  * To enable fMRIPrep to be integrated into a larger processing pipeline, where
+    other tools may generate derivatives that fMRIPrep can use in place of its own
+    steps.
+  * To enable users to substitute their own custom derivatives for those generated
+    by fMRIPrep. For example, a user may wish to use a different brain extraction
+    tool, or a different registration tool, and then use fMRIPrep to generate the
+    remaining derivatives.
+  * To enable complicated meta-workflows, where fMRIPrep is run multiple times
+    with different options, and the results are combined. For example, the
+    `My Connectome <https://openneuro.org/datasets/ds000031/>`__ dataset contains
+    107 sessions for a single-subject. Processing of all sessions simultaneously
+    would be impractical, but the anatomical processing can be done once, and
+    then the functional processing can be done separately for each session.
+
+See also the ``--level`` flag, which can be used to control which derivatives are
+generated.
 
 Troubleshooting
 ---------------
@@ -195,7 +189,7 @@ Information on how to customize and understand these files can be found on the
 page.
 
 **Support and communication**.
-The documentation of this project is found here: https://fmriprep.readthedocs.org/en/latest/.
+The documentation of this project is found here: https://fmriprep.org/en/latest/.
 
 All bugs, concerns and enhancement requests for this software can be submitted here:
 https://github.com/nipreps/fmriprep/issues.

docs/workflows.rst

@@ -37,7 +37,7 @@ is presented below:
      For example, ``anat_preproc_wf`` is a sub-workflow that is generated by the
      :func:`~smriprep.workflows.anatomical.init_anat_preproc_wf` (see below).
      Because each task and run of functional data is processed separately,
-     :func:`~fmriprep.workflows.bold.base.init_func_preproc_wf` names the
+     :func:`~fmriprep.workflows.bold.base.init_bold_wf` names the
      resulting workflows using input parameters, resulting in
      ``func_preproc_task_{task}_run_{run}_wf``.
    * Datasinks begin with ``ds_``, and save files to the output directory.
@@ -287,19 +287,41 @@ from the ``aseg.mgz`` file as described in
 
 BOLD preprocessing
 ------------------
-:py:func:`~fmriprep.workflows.bold.base.init_func_preproc_wf`
+*fMRIPrep* performs a series of steps to preprocess :abbr:`BOLD (blood-oxygen level-dependent)`
+data. Broadly, these are split into fit and transform stages.
+
+The following figures show the overall workflow graph and the ``bold_fit_wf``
+subgraph:
+
+:py:func:`~fmriprep.workflows.bold.base.init_bold_wf`
 
 .. workflow::
     :graph2use: orig
     :simple_form: yes
 
     from fmriprep.workflows.tests import mock_config
     from fmriprep import config
-    from fmriprep.workflows.bold.base import init_func_preproc_wf
+    from fmriprep.workflows.bold.base import init_bold_wf
     with mock_config():
         bold_file = config.execution.bids_dir / 'sub-01' / 'func' \
             / 'sub-01_task-mixedgamblestask_run-01_bold.nii.gz'
-        wf = init_func_preproc_wf(str(bold_file))
+        wf = init_bold_wf(bold_series=[str(bold_file)])
+
+.. _bold_fit:
+
+:py:func:`~fmriprep.workflows.bold.fit.init_bold_fit_wf`
+
+.. workflow::
+    :graph2use: orig
+    :simple_form: yes
+
+    from fmriprep.workflows.tests import mock_config
+    from fmriprep import config
+    from fmriprep.workflows.bold.fit import init_bold_fit_wf
+    with mock_config():
+        bold_file = config.execution.bids_dir / 'sub-01' / 'func' \
+            / 'sub-01_task-mixedgamblestask_run-01_bold.nii.gz'
+        wf = init_bold_fit_wf(bold_series=[str(bold_file)], fieldmap_id="fmap")
 
 Preprocessing of :abbr:`BOLD (blood-oxygen level-dependent)` files is
 split into multiple sub-workflows described below.
@@ -308,31 +330,31 @@ split into multiple sub-workflows described below.
 
 BOLD reference image estimation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-:py:func:`~niworkflows.func.util.init_bold_reference_wf`
+:py:func:`~fmriprep.workflows.bold.reference.init_raw_boldref_wf`
 
 .. workflow::
     :graph2use: orig
     :simple_form: yes
 
-    from niworkflows.func.util import init_bold_reference_wf
-    wf = init_bold_reference_wf(omp_nthreads=1)
+    from fmriprep.workflows.bold.reference import init_raw_boldref_wf
+    wf = init_raw_boldref_wf()
 
 This workflow estimates a reference image for a
-:abbr:`BOLD (blood-oxygen level-dependent)` series.
-If a single-band reference ("sbref") image associated with the BOLD series is
-available, then it is used directly.
-If not, a reference image is estimated from the BOLD series as follows:
+:abbr:`BOLD (blood-oxygen level-dependent)` series as follows:
 When T1-saturation effects ("dummy scans" or non-steady state volumes) are
 detected, they are averaged and used as reference due to their
 superior tissue contrast.
 Otherwise, a median of motion corrected subset of volumes is used.
 
-The reference image is then used to calculate a brain mask for the
-:abbr:`BOLD (blood-oxygen level-dependent)` signal using *NiWorkflows*'
-:py:func:`~niworkflows.func.util.init_enhance_and_skullstrip_bold_wf`.
-Further, the reference is fed to the :ref:`head-motion estimation
-workflow <bold_hmc>` and the :ref:`registration workflow to map
-BOLD series into the T1w image of the same subject <bold_reg>`.
+This reference is used for :ref:`head-motion estimation <bold_hmc>`.
+
+For the :ref:`registration workflow <bold_reg>`, the reference image is
+either the above described reference image or a single-band reference,
+if one is found in the input dataset.
+In either case, this image is contrast-enhanced and skull-stripped
+(see :py:func:`~niworkflows.func.util.init_enhance_and_skullstrip_bold_wf`).
+If fieldmaps are present, the skull-stripped reference is corrected
+prior to registration.
 
 .. figure:: _static/sub-01_task-balloonanalogrisktask_run-1_desc-rois_bold.svg
 
@@ -418,14 +440,19 @@ Theory, methods and references are found within the
 
 Pre-processed BOLD in native space
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-:py:func:`~fmriprep.workflows.bold.resampling.init_bold_preproc_trans_wf`
+:py:func:`~fmriprep.workflows.bold.fit.init_bold_native_wf`
 
 .. workflow::
     :graph2use: orig
     :simple_form: yes
 
-    from fmriprep.workflows.bold import init_bold_preproc_trans_wf
-    wf = init_bold_preproc_trans_wf(mem_gb=3, omp_nthreads=1)
+    from fmriprep.workflows.tests import mock_config
+    from fmriprep import config
+    from fmriprep.workflows.bold.fit import init_bold_native_wf
+    with mock_config():
+        bold_file = config.execution.bids_dir / 'sub-01' / 'func' \
+            / 'sub-01_task-mixedgamblestask_run-01_bold.nii.gz'
+        wf = init_bold_native_wf(bold_series=[str(bold_file)], fieldmap_id='fmap')
 
 A new *preproc* :abbr:`BOLD (blood-oxygen level-dependent)` series is generated
 from the slice-timing corrected or the original data (if
@@ -442,16 +469,14 @@ Interpolation uses a Lanczos kernel.
 
 EPI to T1w registration
 ~~~~~~~~~~~~~~~~~~~~~~~
-:py:func:`~fmriprep.workflows.bold.registration.init_bold_reg_wf`
+:py:func:`~fmriprep.workflows.bold.registration.init_bbreg_wf`
 
 .. workflow::
-    :graph2use: orig
+    :graph2use: hierarchical
     :simple_form: yes
 
-    from fmriprep.workflows.bold import init_bold_reg_wf
-    wf = init_bold_reg_wf(
-        freesurfer=True,
-        mem_gb=1,
+    from fmriprep.workflows.bold.registration import init_bbreg_wf
+    wf = init_bbreg_wf(
         omp_nthreads=1,
         use_bbr=True,
         bold2t1w_dof=9,
@@ -478,23 +503,14 @@ original, affine registration.
 
 Resampling BOLD runs onto standard spaces
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-:py:func:`~fmriprep.workflows.bold.resampling.init_bold_std_trans_wf`
+:py:func:`~fmriprep.workflows.bold.apply.init_bold_volumetric_resample_wf`
 
 .. workflow::
     :graph2use: colored
     :simple_form: yes
 
-    from niworkflows.utils.spaces import SpatialReferences
-    from fmriprep.workflows.bold import init_bold_std_trans_wf
-    wf = init_bold_std_trans_wf(
-        freesurfer=True,
-        mem_gb=3,
-        omp_nthreads=1,
-        spaces=SpatialReferences(
-            spaces=[('MNI152Lin', {}), ('MNIPediatricAsym', {'cohort': '6'})],
-            checkpoint=True),
-        multiecho=False,
-    )
+    from fmriprep.workflows.bold.apply import init_bold_volumetric_resample_wf
+    wf = init_bold_volumetric_resample_wf(metadata={}, fieldmap_id='fmap')
 
 This sub-workflow concatenates the transforms calculated upstream (see
 `Head-motion estimation`_, `Susceptibility Distortion Correction (SDC)`_ --if
@@ -523,7 +539,10 @@ EPI sampled to FreeSurfer surfaces
     wf = init_bold_surf_wf(
         mem_gb=1,
         surface_spaces=['fsnative', 'fsaverage5'],
-        medial_surface_nan=False)
+        medial_surface_nan=False,
+        metadata={},
+        output_dir='.',
+    )
 
 If FreeSurfer processing is enabled, the motion-corrected functional series
 (after single shot resampling to T1w space) is sampled to the
@@ -540,7 +559,7 @@ HCP Grayordinates
 :py:func:`~fmriprep.workflows.bold.resampling.init_bold_fsLR_resampling_wf`
 
 .. workflow::
-    :graph2use: colored
+    :graph2use: orig
     :simple_form: yes
 
     from fmriprep.workflows.bold.resampling import init_bold_fsLR_resampling_wf
@@ -595,6 +614,17 @@ T2*-driven echo combination
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 :py:func:`~fmriprep.workflows.bold.t2s.init_bold_t2s_wf`
 
+.. workflow::
+    :graph2use: colored
+    :simple_form: yes
+
+    from fmriprep.workflows.bold.t2s import init_bold_t2s_wf
+    wf = init_bold_t2s_wf(
+        echo_times=[0.015, 0.030, 0.045],
+        mem_gb=1,
+        omp_nthreads=1,
+    )
+
 If multi-echo :abbr:`BOLD (blood-oxygen level-dependent)` data is supplied,
 this workflow uses the `tedana`_ `T2* workflow`_ to generate an adaptive T2* map
 and optimally weighted combination of all supplied single echo time series.