Add metadata fields from BEP017.

Update outputs.rst

Update output documentation.

Fix *another* bug.

Update outputs.py

Update outputs.py

Update bids.py

Update expected outputs.

Add atlases dataset link.

Add dataset_description.json for atlases.

Use BIDS-URI for correlation matrix.

docs/outputs.rst

@@ -9,23 +9,25 @@ The  *XCP-D* outputs are written out in BIDS format and consist of three main pa
 
 .. admonition:: A note on BIDS compliance
 
-   *XCP-D* attempts to follow the BIDS specification as best as possible.
+   *XCP-D* attempts to follow the BIDS specification as closely as possible.
    However, many *XCP-D* derivatives are not currently covered by the specification.
    In those instances, we attempt to follow recommendations from existing BIDS Extension Proposals
    (BEPs), which are in-progress proposals to add new features to BIDS.
+   However, we do not guarantee compliance with any BEP,
+   as they are not yet part of the official BIDS specification.
 
-   Three BEPs that are of particular use in *XCP-D* are
+   Four BEPs that are of particular use in *XCP-D* are
+   `BEP011: Structural preprocessing derivatives <https://github.com/bids-standard/bids-specification/pull/518>`_,
    `BEP012: Functional preprocessing derivatives <https://github.com/bids-standard/bids-specification/pull/519>`_,
-   `BEP017: BIDS connectivity matrix data schema <https://docs.google.com/document/d/1ugBdUF6dhElXdj3u9vw0iWjE6f_Bibsro3ah7sRV0GA/edit?usp=sharing>`_,
+   `BEP017: Relationship & connectivity matrix data schema <https://docs.google.com/document/d/1ugBdUF6dhElXdj3u9vw0iWjE6f_Bibsro3ah7sRV0GA/edit?usp=sharing>`_,
    and
-   `BEP038: Atlas Specification <https://docs.google.com/document/d/1RxW4cARr3-EiBEcXjLpSIVidvnUSHE7yJCUY91i5TfM/edit?usp=sharing>`_
-   (currently unnumbered).
+   `BEP038: Atlas Specification <https://docs.google.com/document/d/1RxW4cARr3-EiBEcXjLpSIVidvnUSHE7yJCUY91i5TfM/edit?usp=sharing>`_.
 
    In cases where a derivative type is not covered by an existing BEP,
    we have simply attempted to follow the general principles of BIDS.
 
-   If you discover a problem with the BIDS compliance of *XCP-D*'s derivatives, please open an
-   issue in the *XCP-D* repository.
+   If you discover a problem with the BIDS compliance of *XCP-D*'s derivatives,
+   please open an issue in the *XCP-D* repository.
 
 
 ***************
@@ -86,6 +88,7 @@ Atlases are written out to the ``atlases`` subfolder, following BEP038.
 
    xcp_d/
       atlases/
+         dataset_description.json
          atlas-<label>/
             atlas-<label>_dseg.json
             atlas-<label>_dseg.tsv
@@ -155,9 +158,9 @@ atlases it uses to parcellate the functional outputs.
    xcp_d/
       sub-<label>/[ses-<label>/]
          anat/
-            <source_entities>_space-fsLR_seg-<label>_den-32k_desc-curv_morph.tsv
-            <source_entities>_space-fsLR_seg-<label>_den-32k_desc-sulc_morph.tsv
-            <source_entities>_space-fsLR_seg-<label>_den-32k_desc-thickness_morph.tsv
+            <source_entities>_space-fsLR_seg-<label>_den-32k_stat-mean_desc-curv_morph.tsv
+            <source_entities>_space-fsLR_seg-<label>_den-32k_stat-mean_desc-sulc_morph.tsv
+            <source_entities>_space-fsLR_seg-<label>_den-32k_stat-mean_desc-thickness_morph.tsv
 
 
 ******************
@@ -208,15 +211,33 @@ Denoised or residual BOLD data
    and primarily exist for compatibility with DCAN-specific analysis tools.
 
 The sidecar json files contains parameters of the data and processing steps.
+The Sources field contains BIDS URIs pointing to the files used to create the derivative.
+The associated DatasetLinks are defined in the dataset_description.json.
 
    .. code-block:: json-object
 
       {
-         "Freq Band": [0.01, 0.08],
-         "RepetitionTime": 2.0,
-         "compression": true,
-         "dummy vols": 0,
-         "nuisance parameters": "27P",
+         "EchoTime": 0.0424,
+         "EffectiveEchoSpacing": 0.000639989,
+         "FlipAngle": 51,
+         "Manufacturer": "Siemens",
+         "ManufacturersModelName": "Skyra",
+         "NuisanceParameters": "gsr_only",
+         "PhaseEncodingDirection": "j-",
+         "RepetitionTime": 3,
+         "SoftwareFilters": {
+            "Bandpass filter": {
+               "Filter order": 2,
+               "High-pass cutoff (Hz)": 0.01,
+               "Low-pass cutoff (Hz)": 0.08
+            }
+         },
+         "Sources": [
+            "bids:preprocessed:sub-0000001/ses-01/func/sub-0000001_ses-01_task-rest_space-MNI152NLin6Asym_desc-preproc_bold.nii.gz",
+            "bids:xcp_d:sub-0000001/ses-01/func/sub-0000001_ses-01_task-rest_outliers.tsv",
+            "bids:xcp_d:sub-0000001/ses-01/func/sub-0000001_ses-01_task-rest_desc-preproc_design.tsv"
+         ],
+         "TaskName": "resting state"
       }
 
 
@@ -300,13 +321,8 @@ Other outputs include quality control, framewise displacement, and confounds fil
             <source_entities>[_desc-filtered]_motion.tsv
             <source_entities>_outliers.tsv
             <source_entities>_design.tsv
-
-            # NIfTI
             <source_entities>_space-<label>_desc-linc_qc.csv
 
-            # CIFTI
-            <source_entities>_space-fsLR_desc-linc_qc.csv
-
 ``[desc-filtered]_motion.tsv`` is a tab-delimited file with seven columns:
 one for each of the six filtered motion parameters, as well as "framewise_displacement".
 If no motion filtering was applied, this file will not have the ``desc-filtered`` entity.

xcp_d/tests/data/test_ds001419_cifti_outputs.txt

@@ -1,4 +1,5 @@
 xcp_d/atlases
+xcp_d/atlases/dataset_description.json
 xcp_d/atlases/atlas-4S1056Parcels
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.json
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.tsv

xcp_d/tests/data/test_fmriprep_without_freesurfer_outputs.txt

@@ -1,4 +1,5 @@
 xcp_d/atlases
+xcp_d/atlases/dataset_description.json
 xcp_d/atlases/atlas-4S1056Parcels
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.json
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.tsv

xcp_d/tests/data/test_nibabies_outputs.txt

@@ -1,4 +1,5 @@
 xcp_d/atlases
+xcp_d/atlases/dataset_description.json
 xcp_d/atlases/atlas-4S1056Parcels
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.json
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.tsv

xcp_d/tests/data/test_pnc_cifti_outputs.txt

@@ -1,4 +1,5 @@
 xcp_d/atlases
+xcp_d/atlases/dataset_description.json
 xcp_d/atlases/atlas-HCP
 xcp_d/atlases/atlas-HCP/atlas-HCP_dseg.json
 xcp_d/atlases/atlas-HCP/atlas-HCP_dseg.tsv

xcp_d/tests/data/test_pnc_cifti_t2wonly_outputs.txt

@@ -1,4 +1,5 @@
 xcp_d/atlases
+xcp_d/atlases/dataset_description.json
 xcp_d/atlases/atlas-4S156Parcels
 xcp_d/atlases/atlas-4S156Parcels/atlas-4S156Parcels_dseg.json
 xcp_d/atlases/atlas-4S156Parcels/atlas-4S156Parcels_dseg.tsv

xcp_d/tests/data/test_ukbiobank_outputs.txt

@@ -1,4 +1,5 @@
 xcp_d/atlases
+xcp_d/atlases/dataset_description.json
 xcp_d/atlases/atlas-4S1056Parcels
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.json
 xcp_d/atlases/atlas-4S1056Parcels/atlas-4S1056Parcels_dseg.tsv

xcp_d/utils/bids.py

@@ -676,7 +676,7 @@ def collect_run_data(layout, bold_file, cifti, target_space):
     return run_data
 
 
-def write_dataset_description(fmri_dir, xcpd_dir, custom_confounds_folder=None):
+def write_dataset_description(fmri_dir, xcpd_dir, atlases=None, custom_confounds_folder=None):
     """Write dataset_description.json file for derivatives.
 
     Parameters
@@ -685,6 +685,10 @@ def write_dataset_description(fmri_dir, xcpd_dir, custom_confounds_folder=None):
         Path to the BIDS derivative dataset being ingested.
     xcpd_dir : :obj:`str`
         Path to the output xcp-d dataset.
+    atlases : :obj:`list` of :obj:`str`, optional
+        Names of requested XCP-D atlases.
+    custom_confounds_folder : :obj:`str`, optional
+        Path to the folder containing custom confounds files.
     """
     import json
     import os
@@ -737,6 +741,12 @@ def write_dataset_description(fmri_dir, xcpd_dir, custom_confounds_folder=None):
 
     dset_desc["DatasetLinks"]["xcp_d"] = str(xcpd_dir)
 
+    if atlases:
+        if "atlases" in dset_desc["DatasetLinks"].keys():
+            LOGGER.warning("'atlases' is already a dataset link. Overwriting.")
+
+        dset_desc["DatasetLinks"]["atlases"] = os.path.join(xcpd_dir, "atlases")
+
     if custom_confounds_folder:
         if "custom_confounds" in dset_desc["DatasetLinks"].keys():
             LOGGER.warning("'custom_confounds' is already a dataset link. Overwriting.")
@@ -757,6 +767,47 @@ def write_dataset_description(fmri_dir, xcpd_dir, custom_confounds_folder=None):
             json.dump(dset_desc, fo, indent=4, sort_keys=True)
 
 
+def write_atlas_dataset_description(atlas_dir):
+    """Write dataset_description.json file for Atlas derivatives.
+
+    Parameters
+    ----------
+    atlas_dir : :obj:`str`
+        Path to the output XCP-D Atlases dataset.
+    """
+    import json
+    import os
+
+    from xcp_d.__about__ import DOWNLOAD_URL, __version__
+
+    dset_desc = {
+        "Name": "XCP-D Atlases",
+        "DatasetType": "atlas",
+        "GeneratedBy": [
+            {
+                "Name": "xcp_d",
+                "Version": __version__,
+                "CodeURL": DOWNLOAD_URL,
+            },
+        ],
+        "HowToAcknowledge": "Include the generated boilerplate in the methods section.",
+    }
+    os.makedirs(atlas_dir, exist_ok=True)
+
+    atlas_dset_description = os.path.join(atlas_dir, "dataset_description.json")
+    if os.path.isfile(atlas_dset_description):
+        with open(atlas_dset_description, "r") as fo:
+            old_dset_desc = json.load(fo)
+
+        old_version = old_dset_desc["GeneratedBy"][0]["Version"]
+        if Version(__version__).public != Version(old_version).public:
+            LOGGER.warning(f"Previous output generated by version {old_version} found.")
+
+    else:
+        with open(atlas_dset_description, "w") as fo:
+            json.dump(dset_desc, fo, indent=4, sort_keys=True)
+
+
 def get_preproc_pipeline_info(input_type, fmri_dir):
     """Get preprocessing pipeline information from the dataset_description.json file."""
     import json
@@ -1023,6 +1074,20 @@ def _make_xcpd_uri_lol(in_list, output_dir):
     return out_lol
 
 
+def _make_atlas_uri(out_file, output_dir):
+    """Convert postprocessing atlas derivative's path to BIDS URI."""
+    import os
+
+    from xcp_d.utils.bids import _make_uri
+
+    dataset_path = os.path.join(output_dir, "xcp_d", "atlases")
+
+    if isinstance(out_file, list):
+        return [_make_uri(of, "atlas", dataset_path) for of in out_file]
+    else:
+        return [_make_uri(out_file, "atlas", dataset_path)]
+
+
 def _make_preproc_uri(out_file, fmri_dir):
     """Convert preprocessing derivative's path to BIDS URI."""
     from xcp_d.utils.bids import _make_uri

xcp_d/workflows/base.py

@@ -30,6 +30,7 @@
     get_entity,
     get_preproc_pipeline_info,
     group_across_runs,
+    write_atlas_dataset_description,
     write_dataset_description,
 )
 from xcp_d.utils.doc import fill_doc
@@ -196,7 +197,14 @@ def init_xcpd_wf(
     xcpd_wf.base_dir = work_dir
     LOGGER.info(f"Beginning the {name} workflow")
 
-    write_dataset_description(fmri_dir, os.path.join(output_dir, "xcp_d"))
+    write_dataset_description(
+        fmri_dir,
+        os.path.join(output_dir, "xcp_d"),
+        atlases=atlases,
+        custom_confounds_folder=custom_confounds_folder,
+    )
+    if atlases:
+        write_atlas_dataset_description(os.path.join(output_dir, "xcp_d", "atlases"))
 
     for subject_id in subject_list:
         single_subj_wf = init_subject_wf(

xcp_d/workflows/outputs.py

@@ -8,6 +8,7 @@
 from xcp_d.interfaces.bids import DerivativesDataSink
 from xcp_d.interfaces.utils import FilterUndefined
 from xcp_d.utils.bids import (
+    _make_atlas_uri,
     _make_custom_uri,
     _make_preproc_uri,
     _make_xcpd_uri,
@@ -289,7 +290,7 @@ def init_postproc_derivatives_wf(
 
     # Create dictionary of basic information
     cleaned_data_dictionary = {
-        "nuisance parameters": params,
+        "NuisanceParameters": params,
         **source_metadata,
     }
     software_filters = None
@@ -557,7 +558,7 @@ def init_postproc_derivatives_wf(
         )
         workflow.connect([
             (inputnode, make_atlas_dict, [
-                (("atlas_files", _make_xcpd_uri, output_dir), "Sources"),
+                (("atlas_files", _make_atlas_uri, output_dir), "Sources"),
             ]),
         ])  # fmt:skip
 
@@ -655,15 +656,22 @@ def init_postproc_derivatives_wf(
         make_corrs_meta_dict = pe.MapNode(
             niu.Function(
                 function=_make_dictionary,
-                input_names=["Sources"],
+                input_names=["Sources", "NodeFiles"],
                 output_names=["metadata"],
             ),
             run_without_submitting=True,
             mem_gb=1,
             name="make_corrs_meta_dict",
-            iterfield=["Sources"],
+            iterfield=["Sources", "NodeFiles"],
         )
-        workflow.connect([(ds_timeseries, make_corrs_meta_dict, [("out_file", "Sources")])])
+        workflow.connect([
+            (inputnode, make_corrs_meta_dict, [
+                (("atlas_files", _make_atlas_uri, output_dir), "NodeFiles"),
+            ]),
+            (ds_timeseries, make_corrs_meta_dict, [
+                (("out_file", _make_xcpd_uri, output_dir), "Sources"),
+            ]),
+        ])  # fmt:skip
 
         ds_correlations = pe.MapNode(
             DerivativesDataSink(
@@ -674,6 +682,12 @@ def init_postproc_derivatives_wf(
                 statistic="pearsoncorrelation",
                 suffix="relmat",
                 extension=".tsv",
+                # Metadata
+                RelationshipMeasure="Pearson correlation coefficient",
+                Weighted=True,
+                Directed=False,
+                ValidDiagonal=False,
+                StorageFormat="Full",
             ),
             name="ds_correlations",
             run_without_submitting=True,
@@ -762,19 +776,22 @@ def init_postproc_derivatives_wf(
             make_ccorrs_meta_dict = pe.MapNode(
                 niu.Function(
                     function=_make_dictionary,
-                    input_names=["Sources"],
+                    input_names=["Sources", "NodeFiles"],
                     output_names=["metadata"],
                 ),
                 run_without_submitting=True,
                 mem_gb=1,
                 name="make_ccorrs_meta_dict",
-                iterfield=["Sources"],
+                iterfield=["Sources", "NodeFiles"],
             )
-            # fmt:off
             workflow.connect([
-                (ds_timeseries_ciftis, make_ccorrs_meta_dict, [("out_file", "Sources")]),
-            ])
-            # fmt:on
+                (inputnode, make_ccorrs_meta_dict, [
+                    (("atlas_files", _make_atlas_uri, output_dir), "NodeFiles"),
+                ]),
+                (ds_timeseries_ciftis, make_ccorrs_meta_dict, [
+                    (("out_file", _make_xcpd_uri, output_dir), "Sources"),
+                ]),
+            ])  # fmt:skip
 
             ds_correlation_ciftis = pe.MapNode(
                 DerivativesDataSink(
@@ -787,6 +804,12 @@ def init_postproc_derivatives_wf(
                     statistic="pearsoncorrelation",
                     suffix="boldmap",
                     extension=".pconn.nii",
+                    # Metadata
+                    RelationshipMeasure="Pearson correlation coefficient",
+                    Weighted=True,
+                    Directed=False,
+                    ValidDiagonal=False,
+                    StorageFormat="Full",
                 ),
                 name="ds_correlation_ciftis",
                 run_without_submitting=True,
@@ -796,7 +819,9 @@ def init_postproc_derivatives_wf(
             ds_correlation_ciftis.inputs.segmentation = atlases
             # fmt:off
             workflow.connect([
-                (inputnode, ds_correlation_ciftis, [("correlation_ciftis", "in_file")]),
+                (inputnode, ds_correlation_ciftis, [
+                    ("correlation_ciftis", "in_file"),
+                ]),
                 (make_ccorrs_meta_dict, ds_correlation_ciftis, [("metadata", "meta_dict")]),
             ])
             # fmt:on