More updates to documentation

* Use literalincludes almost exclusively within tutorial to make
  maintenance easier
* Update non-tutorial documentation files to 1.0 syntax
* Add more links to attributes like zip_lists and path
* Update typing annotations within source code for more consistency, and
  to take advantage of auto-documentation.
* Add instructions to download example dataset
* Various formatting fixes

docs/api.rst

@@ -2,6 +2,8 @@
 API
 ================
 
+
+
 snakebids
 ---------
 
@@ -20,3 +22,22 @@ snakemake_io
 
 .. automodule:: snakebids.utils.snakemake_io
     :members:
+
+
+cli
+---
+
+.. automodule:: snakebids.cli
+    :members:
+
+exceptions
+----------
+
+.. automodule:: snakebids.exceptions
+    :members:
+
+types
+-----
+
+.. automodule:: snakebids.types
+    :members:

docs/bids_app/config.md

@@ -1,3 +1,4 @@
+{#bids-app-config}
 Configuration
 =============
 
@@ -21,10 +22,13 @@ The value of ``filters`` should be a dictionary where each key corresponds to a
         extension: '.nii.gz'
         datatype: 'func'
   ```
+
   the bold component would match any paths under the `func/` datatype folder, with the suffix `bold` and the extension `.nii.gz`.
+
   ```
   sub-xxx/.../func/ent1-xxx_ent2-xxx_..._bold.nii.gz
   ```
+
 * [boolean](python:bool): constrains presence or absence of the entity without restricting its value. `False` requires that the entity be **absent**, while `True` requires that the  entity be **present**, regardless of value.
   ```yaml
   pybids_inputs:
@@ -38,7 +42,7 @@ The value of ``filters`` should be a dictionary where each key corresponds to a
 
 In addition, the special filter `regex_search` can be set to `true`, which causes all other filters in the component to use regex matching instead of exact matching.
 
-The value of ``wildcards`` should be a list of BIDS entities. Snakebids collects the values of any entities specified and saves them in the {attr}`entities <snakebids.BidsComponent.entities>` and `zip_lists` entries of the corresponding {class}`BidsComponent <snakebids.BidsComponent>`. In other words, these are the entities to be preserved in output paths derived from the input being described. Placing an entity in `wildcards` does not require the entity be present. If an entity is not found, it will be left out of {attr}`entities <snakebids.BidsComponent.entities>`. To require the presence of an entity, place it under `filters` set to `true`.
+The value of ``wildcards`` should be a list of BIDS entities. Snakebids collects the values of any entities specified and saves them in the {attr}`entities <snakebids.BidsComponent.entities>` and {attr}`~snakebids.BidsComponent.zip_lists` entries of the corresponding {class}`BidsComponent <snakebids.BidsComponent>`. In other words, these are the entities to be preserved in output paths derived from the input being described. Placing an entity in `wildcards` does not require the entity be present. If an entity is not found, it will be left out of {attr}`entities <snakebids.BidsComponent.entities>`. To require the presence of an entity, place it under `filters` set to `true`.
 
 In the following (YAML-formatted) example, the ``bold`` input type is specified. BIDS files with the datatype ``func``, suffix ``bold``, and extension ``.nii.gz`` will be grabbed, and the ``subject``, ``session``, ``acquisition``, ``task``, and ``run`` entities of those files will be left as wildcards. The `task` entity must be present, but there must not be any `desc`.
 
@@ -61,24 +65,23 @@ pybids_inputs:
 
 ### `pybids_db_dir`
 
-PyBIDS allows for the use of a cached layout to be used in order to reduce the time required to index a BIDS dataset. A path (if provided) to save the ``PyBIDS`` layout. If ``None`` or ``''`` is provided, the layout is not saved / used. The path provided must be absolute, otherwise the database will not be used. Note, this is a variable used for an opt-in feature and must first be uncommented in the ``snakebids.yml`` file.
+PyBIDS allows for the use of a cached layout to be used in order to reduce the time required to index a BIDS dataset. A path (if provided) to save the *pybids* [layout](#bids.layout.BIDSLayout). If `None` or `''` is provided, the layout is not saved or used. The path provided must be absolute, otherwise the database will not be used.
 
 ### `pybids_db_reset`
-A boolean determining whether the existing layout should be be updated. Default behaviour does not update the existing database if one is used. Note, this is a variable used for an opt-in feature and must first be uncommented in the ``snakebids.yml`` file.
+
+A boolean determining whether the existing layout should be be updated. Default behaviour does not update the existing database if one is used.
 
 ### `analysis_levels`
 
 A list of analysis levels in the BIDS app. Typically, this will include participant and/or group. Note that the default (YAML) configuration file expects this mapping to be identified with the anchor ``analysis_levels`` to be aliased by ``parse_args``.
 
-
 ### `targets_by_analysis_level`
 
 A mapping from the name of each ``analysis_level`` to the list of rules or files to be run for that analysis level.
 
-
 ### `parse_args`
 
-A dictionary of command-line parameters to make available as part of the BIDS app. Each item of the mapping is passed to [argparse's add_argument function](https://docs.python.org/3/library/argparse.html#the-add-argument-method). A number of default entries are present in a new snakebids project's config file that structure the BIDS app's CLI, but additional command-line arguments can be added as necessary.
+A dictionary of command-line parameters to make available as part of the BIDS app. Each item of the mapping is passed to [argparse's add_argument function](#argparse.ArgumentParser.add_argument). A number of default entries are present in a new snakebids project's config file that structure the BIDS app's CLI, but additional command-line arguments can be added as necessary.
 
 
 ### `debug`

docs/bids_app/workflow.md

@@ -0,0 +1,57 @@
+Workflows
+=========
+
+Snakebids workflows are constructed the same way as any other [Snakemake workflows](inv:snakemake#snakefiles/rules), but with a few additions that make it easier to work with BIDS datasets.
+
+To get access to these additions, the base Snakefile for a snakebids workflow should begin with the following boilerplate:
+
+```python
+import snakebids
+from snakebids import bids
+
+configfile: 'config/snakebids.yml'
+
+# Get input wildcards
+inputs = snakebids.generate_inputs(
+    bids_dir=config["bids_dir"],
+    pybids_inputs=config["pybids_inputs"],
+    pybids_database_dir=config.get("pybids_db_dir"),
+    pybids_reset_database=config.get("pybids_db_reset"),
+    derivatives=config.get("derivatives"),
+    participant_label=config.get("participant_label"),
+    exclude_participant_label=config.get("exclude_participant_label"),
+    use_bids_inputs=True,
+)
+
+```
+
+Snakebids workflow features
+---------------------------
+
+The [](#snakebids.bids) function generates a properly-formatted BIDS filename with the specified entities, as documented in more detail elsewhere in this documentation.
+
+[](#snakebids.generate_inputs) returns an instance of [](#snakebids.BidsDataset), a special [](#dict) with keys mapping to the {class}`~snakebids.BidsComponents` defined in [the config file](/bids_app/config). Each {class}`~snakebids.BidsComponent` contains a number of attributes to assist processing a BIDS dataset with snakemake. {func}`~snakebids.generate_inputs` should be called at the beginning of the workflow and assigned to a variable called `inputs`.
+
+The {attr}`~snakebids.BidsComponent.path` member of [](#snakebids.BidsComponent) is generated by snakebids and contains a list of matched files for every input type. Often, the first rule to be invoked will use one or more entries in `inputs.path` as the input file specification.
+
+The {attr}`~snakebids.BidsComponent.zip_lists` member of [](#snakebids.BidsComponent) is used with [`bids()`](#snakebids.bids) and [`expand()`](#snakefiles_expand) to fill the wildcards with corresponding values from input files. The usage pattern is as follows:
+
+```py
+expand(
+    bids(
+        root="results",
+        datatype="func",
+        suffix="bold.nii.gz",
+        **inputs.wildcards["bold"]
+    ),
+    zip,
+    **inputs.zip_lists["bold"]
+)
+```
+
+The {attr}`~snakebids.BidsComponent.wildcards` member of {class}`snakebids.BidsComponent` is generated by snakebids and contains a dictionary mapping the wildcards for each input type to snakemake-formatted wildcards, for convenient use in the ``bids`` function.
+
+
+## Accessing the underlying *pybids* dataset
+
+In addition to mapping all of the {class}`~snakebids.BidsComponents` to their names, {class}`~snakebids.BidsDataset` also has a {attr}`~snakebids.BidsDataset.layout` member which gives access to the underlying {class}`BIDSLayout <bids.layout.BIDSLayout>`. This can be used to access advanced pybids features not covered by `snakebids`. Note that if `custom_paths` are specified for every {class}`BidsComponent <snakebids.BidsComponent>`, pybids indexing will be skipped and `layout` will be set to `None`. If your workflow relies on accessing this `layout`, you must ensure your users do not provide a `custom_path` for every single component, either in the config file or [via the CLI](/running_snakebids/overview) (``--path_{component}``).

docs/conf.py

@@ -34,16 +34,18 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    "sphinx_rtd_theme",
     "sphinxarg.ext",
     "sphinx.ext.intersphinx",
     "sphinx.ext.napoleon",
     "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
     "sphinxcontrib.asciinema",
     "myst_parser",
     "sphinx_copybutton",
+    "sphinx_toolbox.more_autosummary",
 ]
 
+
 myst_enable_extensions = [
     "attrs_block",
 ]
@@ -60,6 +62,10 @@
 napoleon_google_docstring = False
 napoleon_numpy_docstring = True
 
+autodoc_member_order = "bysource"
+autodoc_typehints = "description"
+autosummary_imported_members = True
+
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
@@ -90,6 +96,7 @@
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
+# templates_path = ["_templates"]
 
 sphinxcontrib_asciinema_defaults = {
     "preload": 1,

docs/requirements.txt

@@ -1,6 +1,6 @@
-docutils<0.18
+# docutils<0.18
 sphinx-argparse
-sphinx_rtd_theme
+# sphinx_rtd_theme
 sphinxcontrib-asciinema
 myst-parser
 furo==2023.3.23

docs/running_snakebids/overview.md

@@ -1,5 +1,3 @@
-(running overview)=
-
 Overview
 ========
 

docs/tutorial/step0/Snakefile

@@ -1,6 +1,6 @@
 rule smooth:
     input:
-        '../bids/sub-001/func/sub-001_task-rest_run-1_bold.nii.gz'
+        'data/sub-001/func/sub-001_task-rest_run-1_bold.nii.gz'
     params:
         sigma = '2.12'
     output:

docs/tutorial/step1/Snakefile

@@ -1,6 +1,6 @@
 rule smooth:
     input:
-        '../bids/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
+        'data/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
     params:
         sigma = '2.12'
     output:

docs/tutorial/step2/Snakefile

@@ -3,7 +3,7 @@ def calc_sigma_from_fwhm(wildcards):
 
 rule smooth:
     input:
-        '../bids/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
+        'data/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
     params:
         sigma = calc_sigma_from_fwhm
     output:

docs/tutorial/step3/Snakefile

@@ -14,7 +14,7 @@ def calc_sigma_from_fwhm(wildcards):
 
 rule smooth:
     input:
-        '../bids/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
+        'data/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
     params:
         sigma = calc_sigma_from_fwhm,
     output:

docs/tutorial/step4/config.yml

@@ -15,4 +15,4 @@ fwhm:
   - 20
 
 
-in_bold: '../bids/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
+in_bold: 'data/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'

docs/tutorial/step5/Snakefile

@@ -29,11 +29,13 @@ rule smooth:
     params:
         sigma = calc_sigma_from_fwhm
     output:
-        bids(root='results',
-                subject='{subject}',
-                task='{task}',
-                run='{run}',
-                fwhm='{fwhm}',
-                suffix='bold.nii.gz')
+        bids(
+            root='results',
+            subject='{subject}',
+            task='{task}',
+            run='{run}',
+            fwhm='{fwhm}',
+            suffix='bold.nii.gz',
+        )
     shell:
         'fslmaths {input} -s {params.sigma} {output}'

docs/tutorial/step5/config.yml

@@ -15,4 +15,4 @@ fwhm:
   - 20
 
 
-in_bold: '../bids/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'
+in_bold: 'data/sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.nii.gz'

docs/tutorial/step6/Snakefile

@@ -1,46 +1,48 @@
-import snakebids
-from snakebids import bids
+from snakebids import bids, generate_inputs
 
 configfile: 'config.yml'
 
-config.update(
-    snakebids.generate_inputs(
-        bids_dir=config['bids_dir'],
-        pybids_inputs=config['pybids_inputs'],
-    )
+inputs = generate_inputs(
+    bids_dir=config['bids_dir'],
+    pybids_inputs=config['pybids_inputs'],
+    use_bids_inputs=True,
 )
 
-import pprint
-pprint.pp(config)
+print(inputs)
 
 rule all:
     input:
-        expand(bids(root='results',
+        expand(
+            bids(root='results',
                 subject='{subject}',
                 task='{task}',
                 run='{run}',
                 fwhm='{fwhm}',
-                suffix='bold.nii.gz'),
-                    subject=config['subjects'],
-                    task=config['tasks'],
-                    run=config['runs'],
-                    fwhm=config['fwhm'])
+                suffix='bold.nii.gz'
+            ),
+            subject=config['subjects'],
+            task=config['tasks'],
+            run=config['runs'],
+            fwhm=config['fwhm']
+        )
 
 
 def calc_sigma_from_fwhm(wildcards):
     return f'{float(wildcards.fwhm)/2.355:0.2f}'
 
 rule smooth:
     input:
-        config['path']['bold']
+        inputs['bold'].path
     params:
         sigma = calc_sigma_from_fwhm
     output:
-        bids(root='results',
-                subject='{subject}',
-                task='{task}',
-                run='{run}',
-                fwhm='{fwhm}',
-                suffix='bold.nii.gz')
+        bids(
+            root='results',
+            subject='{subject}',
+            task='{task}',
+            run='{run}',
+            fwhm='{fwhm}',
+            suffix='bold.nii.gz'
+        )
     shell:
         'fslmaths {input} -s {params.sigma} {output}'