Code Base Update (#421)

* rename ndmg to m2g * strip down requirements * update to run on python3 not on specific version * add lecture * reorg tutorials * update readme * Update docs * add disc data * Figure 3 * Update * black * figure 4 * Figure 4 data * Header for figure 4 * runner * add description * Change depth * Change title * finish diffusion * update pipelines * Update * finish pipeline * remove files * add * Try new dockerfile * Remove unused imports * update * update * update * update * update * Add setup.cfg for update compatibility * Remove dockerfile
neurodata · Apr 10, 2024 · 6e6cb22 · 6e6cb22
1 parent 59f9519
commit 6e6cb22
Show file tree

Hide file tree

Showing 60 changed files with 46,796 additions and 1,042 deletions.
diff --git a/README.md b/README.md
diff --git a/docs/Makefile b/docs/Makefile
diff --git a/docs/README.md b/docs/README.md
diff --git a/docs/_static/diff_mapped_atlas.png b/docs/_static/diff_mapped_atlas.png
diff --git a/docs/_static/diff_skullstip.png b/docs/_static/diff_skullstip.png
diff --git a/docs/_static/func_motion_plot.png b/docs/_static/func_motion_plot.png
diff --git a/docs/_static/func_skullstrip.png b/docs/_static/func_skullstrip.png
diff --git a/docs/_static/m2g_pipeline.png b/docs/_static/m2g_pipeline.png
diff --git a/docs/_static/qa-d/connectome.png b/docs/_static/qa-d/connectome.png
diff --git a/docs/_static/qa-d/skullstrip.png b/docs/_static/qa-d/skullstrip.png
diff --git a/docs/_static/qa-d/tractography.png b/docs/_static/qa-d/tractography.png
diff --git a/docs/_static/qa-f/func_motion_plot.png b/docs/_static/qa-f/func_motion_plot.png
diff --git a/docs/_static/qa-f/func_skullstrip.png b/docs/_static/qa-f/func_skullstrip.png
diff --git a/docs/conf.py b/docs/conf.py
@@ -53,6 +53,9 @@
     # "toctree_filter",
 ]
 
+# nbxphinx
+nbsphinx_execute = "never"
+
 # -- numpydoc
 # Below is needed to prevent errors
 numpydoc_show_class_members = False

diff --git a/docs/diffusion.rst b/docs/diffusion.rst
diff --git a/docs/docker.rst b/docs/docker.rst
@@ -0,0 +1,39 @@
+.. include:: links.rst
+
+
+Docker
+------
+
+You can learn more about Docker and how to install it in the `Docker installation`_ documentation.
+Please make sure you follow the Docker installation instructions. You can check your Docker Runtime installation running their ``hello-world`` image:
+
+.. code-block:: bash
+
+    $ docker run --rm hello-world
+
+If you have a functional installation, then you should obtain the following output:
+
+.. code-block::
+
+    Hello from Docker!
+    This message shows that your installation appears to be working correctly.
+
+    To generate this message, Docker took the following steps:
+    1. The Docker client contacted the Docker daemon.
+    2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
+        (amd64)
+    3. The Docker daemon created a new container from that image which runs the
+        executable that produces the output you are currently reading.
+    4. The Docker daemon streamed that output to the Docker client, which sent it
+        to your terminal.
+
+    To try something more ambitious, you can run an Ubuntu container with:
+    $ docker run -it ubuntu bash
+
+    Share images, automate workflows, and more with a free Docker ID:
+    https://hub.docker.com/
+
+    For more examples and ideas, visit:
+    https://docs.docker.com/get-started/
+
+After checking your Docker Engine is capable of running Docker images, you are ready to pull your ``m2g`` container image.
diff --git a/docs/functional.rst b/docs/functional.rst
diff --git a/docs/index.rst b/docs/index.rst
@@ -20,17 +20,18 @@ Join us on `GitHub <https://github.com/neurodata/m2g>`_.
 
 .. toctree::
    :caption: Documentation
-   :maxdepth: 3
+   :maxdepth: 1
 
    install
+   usage
    diffusion
    functional
+   paper/index
 
 .. toctree::
    :maxdepth: 1
    :caption: Useful Links
 
    m2g @ GitHub <http://www.github.com/neurodata/m2g/>
    m2g @ PyPI <https://pypi.org/project/m2g/>
-   Issue Tracker <https://github.com/neurodata/m2g/issues>
-   NeuroData Lab <http://neurodata.io/>
+   m2g lecture <https://neurodata.io/talks/ndmg.pdf>
diff --git a/docs/install.rst b/docs/install.rst
@@ -1,177 +1,109 @@
-******************
-Install
-******************
+.. include:: links.rst
+
+------------
+Installation
+------------
 
 .. contents:: Table of Contents
 
+There are two ways to install **m2g**:
 
-Summary
-===================
+* using container technologies - `Docker Container`_ - (RECOMMENDED) or
+* within a Manually Prepared Environment (For ``m2g-d`` pipeline).
 
-Pull docker container::
+However, the manually prepared environment is not recommended, as it is more complex and error-prone.
 
-    docker pull neurodata/m2g
 
-Run dmri participant pipeline::
+Docker Container
+================================================
+**m2g** is distributed as a Docker image, which is the recommended way to run the pipeline. See :doc:`Docker <./docker>`.
 
-    docker run -ti -v /path/to/local/data:/data neurodata/m2g /data/ /data/outputs
+The Docker image contains all the necessary software dependencies, and the pipeline is executed in a containerized environment.
+This ensures that the pipeline runs in a consistent environment, regardless of the host system.
 
-System Requirements
-====================
-.. TODO: update package versions
+The most recent docker image can be pulled using::
+
+    $ docker pull neurodata/m2g:latest
+
+The image can then be used to create a container and run directly with the following command (and any additional options you may require for Docker, such as volume mounting)::
+
+    $ docker run -ti --entrypoint /bin/bash neurodata/m2g:latest
+
+**m2g** docker containers can also be made from m2g's Dockerfile::
+
+    $ git clone https://github.com/neurodata/m2g.git
+    $ cd m2g
+    $ docker build -t <imagename:uniquelabel> .
+
+Where "uniquelabel" can be whatever you wish to call this Docker image (for example, `m2g:latest`).
+Additional information about building Docker images can be found `here <https://docs.docker.com/engine/reference/commandline/image_build/>`_.
+Creating the Docker image should take several minutes if this is the first time you have used this docker file.
+In order to create a docker container from the docker image and access it, use the following command to both create and enter the container::
 
-The m2g pipeline was developed and tested primarily on Mac OSX, Ubuntu (16, 18, 20, 22), and CentOS (5, 6);
+    $ docker run -it --entrypoint /bin/bash m2g:uniquelabel
 
-Made to work on Python 3.8;
+Manually Prepared Environment (For ``m2g-d`` pipeline)
+=======================================================
 
-Is wrapped in a Docker container;
+.. warning::
 
-Has install instructions via a Dockerfile;
+   This method is not recommended! Please consider using containers.
 
-Requires no non-standard hardware to run;
+.. warning::
 
-Has key features built upon FSL, Dipy, Nibabel, Nilearn, Networkx, Numpy, Scipy, Scikit-Learn, and others;
+   Without Docker, you can only run ``m2g-d`` portion of the pipeline. ``m2g-f`` requires CPAC, which also runs
+   on a Docker container.
 
-Takes approximately 1-core, 8-GB of RAM, and 1 hour to run for most datasets.
-
-While m2g is quite robust to Python package versions (with only few exceptions, mentioned in the installation guide), an example of possible versions (taken from the m2g Docker Image with version v0.3.0) is shown below. Note: this list excludes many libraries which are standard with a Python distribution, and a complete list with all packages and versions can be produced by running pip freeze within the Docker container mentioned above. ::
-
-    awscli==1.16.210 , boto3==1.9.200 , botocore==1.12.200 , colorama==0.3.9 , configparser>=3.7.4 ,
-    Cython==0.29.13 , dipy==0.16.0 , duecredit==0.7.0 , fury==0.3.0 , graspy==0.0.3 , ipython==7.7.0 ,
-    matplotlib==3.1.1 , networkx==2.3 , nibabel==2.5.0 , nilearn==0.5.2 , numpy==1.17.0 , pandas==0.25.0,
-    Pillow==6.1.0 , plotly==1.12.9, pybids==0.6.4 , python-dateutil==2.8.0 , PyVTK==0.5.18 ,
-    requests==2.22.0 , s3transfer==0.2.1 , setuptools>=40.0 scikit-image==0.13.0 , scikit-learn==0.21.3 ,
-    scipy==1.3.0 , sklearn==8.0 , vtk==8.1.2
-
-Installation Guide
-==================
-.. TODO: add links to external packages
-
-Currently, the Docker image is recommended.
-
-``pip`` and Github installations are also available.
-
-Docker
---------------
-.. _Dockerhub : https://hub.docker.com/r/neurodata/m2g/
-.. _documentation : https://docs.docker.com/
-
-The neurodata/m3r-release Docker container enables users to run end-to-end connectome estimation on structural MRI or functional MRI right from container launch. The pipeline requires that data be organized in accordance with the BIDS spec. If the data you wish to process is available on S3 you simply need to provide your s3 credentials at build time and the pipeline will auto-retrieve your data for processing.
-
-If you have never used Docker before, it is useful to run through the Docker documentation_.
-
-**Getting Docker container**::
-
-    $ docker pull neurodata/m2g
 
-*(A) I do not wish to use S3:*
+Make sure all of **m2g**'s `External Dependencies`_ are installed.
+These tools must be installed and their binaries available in the
+system's ``$PATH``.
+A relatively interpretable description of how your environment can be set-up
+is found in the `Dockerfile <https://github.com/neurodata/m2g/blob/deploy/Dockerfile>`_.
 
-You are good to go!
+On a functional Python 3.8 (or above) environment with ``pip`` installed,
+**m2g** can be installed using the habitual command ::
 
-*(B) I wish to use S3:*
+    $ python -m pip install m2g
 
-Add your secret key/access id to a file called credentials.csv in this directory on your local machine. A dummy file has been provided to make the format we expect clear. (This is how AWS provides credentials)
+Check your installation with the ``--version`` argument ::
+
+    $ m2g --version
+
+
+External Dependencies
+---------------------
+
+**m2g** requires other neuroimaging software that are not handled by the Python's packaging system (Pypi):
+
+- FSL_ (version 6.0.6.5)
+- ANTs_ (version 2.4.3)
+- AFNI_ (version 23.3.09)
+- `C3D <https://sourceforge.net/projects/c3d/>`_ (version 1.3.0)
+
+
+Requirements
+====================
 
-**Processing Data**
+Hardware Requirements
+---------------------
 
-Below is the help output generated by running **m2g** with the ``-h`` command. All parameters are explained in this output. ::
+The pipeline only requires 1-core and 16-GB of RAM, and takes approximately 1 hour to run for most datasets.
 
-    $ docker run -ti neurodata/m2g -h
 
-    usage: m2g_bids [-h]
-                 [--participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]
-                 [--session_label SESSION_LABEL [SESSION_LABEL ...]]
-                 [--run_label RUN_LABEL [RUN_LABEL ...]] [--bucket BUCKET]
-                 [--remote_path REMOTE_PATH] [--push_data] [--dataset DATASET]
-                 [--atlas ATLAS] [--debug] [--sked] [--skreg] [--vox VOX] [-c]
-                 [--mod MOD] [--tt TT] [--mf MF] [--sp SP] [--seeds SEEDS]
-                 [--modif MODIF]
-                 bids_dir output_dir
+Python Requirements
+-------------------
 
-    This is an end-to-end connectome estimation pipeline from M3r Images.
+The m2g pipeline was developed and tested for Python <=3.8 and <=3.10 on linux systems, such as Ubuntu, CentOS and macOS.
+With `Docker execution`_, **m2g** can run on almost all systems that support Docker.
 
-    positional arguments:
-    bids_dir              The directory with the input dataset formatted
-                          according to the BIDS standard.
-    output_dir            The directory where the output files should be stored.
-                          If you are running group level analysis this folder
-                          should be prepopulated with the results of the
-                          participant level analysis.
+While m2g is quite robust to Python package versions (with only few exceptions, mentioned in the installation guide), an example of possible versions (taken from the m2g Docker Image with version v0.3.0) is shown below. ::
 
-    optional arguments:
-    -h, --help            show this help message and exit
-    --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
-                          The label(s) of the participant(s) that should be
-                          analyzed. The label corresponds to
-                          sub-<participant_label> from the BIDS spec (so it does
-                          not include "sub-"). If this parameter is not provided
-                          all subjects should be analyzed. Multiple participants
-                          can be specified with a space separated list.
-    --session_label SESSION_LABEL [SESSION_LABEL ...]
-                          The label(s) of the session that should be analyzed.
-                          The label corresponds to ses-<participant_label> from
-                          the BIDS spec (so it does not include "ses-"). If this
-                          parameter is not provided all sessions should be
-                          analyzed. Multiple sessions can be specified with a
-                          space separated list.
-    --run_label RUN_LABEL [RUN_LABEL ...]
-                          The label(s) of the run that should be analyzed. The
-                          label corresponds to run-<run_label> from the BIDS
-                          spec (so it does not include "task-"). If this
-                          parameter is not provided all runs should be analyzed.
-                          Multiple runs can be specified with a space separated
-                          list.
-    --bucket BUCKET       The name of an S3 bucket which holds BIDS organized
-                          data. You must have built your bucket with credentials
-                          to the S3 bucket you wish to access.
-    --remote_path REMOTE_PATH
-                          The path to the data on your S3 bucket. The data will
-                          be downloaded to the provided bids_dir on your
-                          machine.
-    --push_data           flag to push derivatives back up to S3.
-    --dataset DATASET     The name of the dataset you are perfoming QC on.
-    --atlas ATLAS         The atlas being analyzed in QC (if you only want one).
-    --debug               If False, remove any old files in the output
-                          directory.
-    --sked                Whether to skip eddy correction if it has already been
-                          run.
-    --skreg               whether or not to skip registration
-    --vox VOX             Voxel size to use for template registrations (e.g.
-                          default is '2mm')
-    -c, --clean           Whether or not to delete intemediates
-    --mod MOD             Determinstic (det) or probabilistic (prob) tracking.
-                          Default is det.
-    --tt TT               Tracking approach: local or particle. Default is
-                        local.
-    --mf MF               Diffusion model: csd or csa. Default is csd.
-    --sp SP               Space for tractography. Default is native.
-    --seeds SEEDS         Seeding density for tractography. Default is 20.
-    --modif MODIF         Name of folder on s3 to push to. If empty, push to a
-                          folder with m2g's version number.
-
-In order to share data between our container and the rest of our machine in Docker, we need to mount a volume. Docker does this with the -v flag. Docker expects its input formatted as: ``-v path/to/local/data:/path/in/container``. We'll do this when we launch our container, as well as give it a helpful name so we can locate it later on.
-
-**To run m2g on data** ::
-
-    docker run -ti -v /path/to/local/data:/data neurodata/m2g /data/ /data/outputs
-
-
-Pip
--------------
-
-m2g relies on FSL, Dipy, networkx, and nibabel, numpy scipy, scikit-learn, scikit-image, nilearn. You should install FSL through the instructions on their website, then follow install other Python dependencies with the following::
-
-    pip install m2g
-
-The only known packages which require a specific version are plotly and networkx, due to backwards-compatability breaking changes.
-
-Installation shouldn't take more than a few minutes, but depends on your internet connection.
-
-Github
------------
-
-To install directly from Github, run::
-
-    git clone https://github.com/neurodata/m2g
-    cd m2g
-    python setup.py install
+    boto3==1.28.4
+    configparser>=3.7.4
+    dipy==0.16.0
+    graspologic>=3.3.0
+    networkx==2.3
+    nibabel==2.5.0
+    nilearn==0.5.2
+    numpy==1.17.0
diff --git a/docs/links.rst b/docs/links.rst
@@ -0,0 +1,23 @@
+.. _Nipype: https://nipype.readthedocs.io/en/latest/
+.. _BIDS: https://bids.neuroimaging.io/
+.. _`BIDS Derivatives`: https://bids-specification.readthedocs.io/en/stable/05-derivatives/01-introduction.html
+.. _Installation: installation.html
+.. _workflows: workflows.html
+.. _FSL: https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/
+.. _ANTs: https://stnava.github.io/ANTs/
+.. _FreeSurfer: https://surfer.nmr.mgh.harvard.edu/
+.. _`submillimeter reconstruction`: https://surfer.nmr.mgh.harvard.edu/fswiki/SubmillimeterRecon
+.. _`mri_robust_template`: https://surfer.nmr.mgh.harvard.edu/fswiki/mri_robust_template
+.. _AFNI: https://afni.nimh.nih.gov/
+.. _GIFTI: https://www.nitrc.org/projects/gifti/
+.. _`Connectome Workbench`: https://www.humanconnectome.org/software/connectome-workbench.html
+.. _`HCP Pipelines`: https://humanconnectome.org/software/hcp-mr-pipelines/
+.. _`Docker Engine`: https://www.docker.com/products/container-runtime
+.. _`Docker installation`: https://docs.docker.com/install/
+.. _`Docker Hub`: https://hub.docker.com/r/nipreps/fmriprep/tags
+.. _Singularity: https://github.com/singularityware/singularity
+..  _SPM: https://www.fil.ion.ucl.ac.uk/spm/software/spm12/
+.. _TACC: https://www.tacc.utexas.edu/
+.. _tedana: https://github.com/me-ica/tedana
+.. _`T2* workflow`: https://tedana.readthedocs.io/en/latest/generated/tedana.workflows.t2smap_workflow.html#tedana.workflows.t2smap_workflow  # noqa
+.. _`citation boilerplate`: https://www.nipreps.org/intro/transparency/#citation-boilerplates