📚 C-PAC Python package (#219)

.circleci/config.yml

@@ -16,11 +16,29 @@ jobs:
             pip install --user -r https://raw.githubusercontent.com/FCP-INDI/C-PAC/develop/requirements.txt
             pip install --user git+https://github.com/FCP-INDI/C-PAC.git sphinx m2r numpydoc PyGithub sphinxcontrib-programoutput
             git clone https://github.com/FCP-INDI/C-PAC.git /build/C-PAC
+            python -m venv ~/simple
+            source ~/simple/bin/activate
+            pip install cpac
+            deactivate
+      - setup_remote_docker
       - run:
-          name: Build docs
+          name: Run cpac commands
           command: |
-            ./bin/build
-
+            source ~/simple/bin/activate
+            cpac run --help
+            mkdir -p docs/user/_sources/cpac
+            printf ".. code-block:: console\n\n   $ cpac --help\n\n" > docs/user/_sources/cpac/help.txt
+            cpac --help | sed -e "s/.*/   &/" >> docs/user/_sources/cpac/help.txt
+            mkdir -p docs/user/_sources/run
+            printf "Usage: cpac run\n\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\n.. code-block:: console\n\n   $ cpac run --help\n\n" > docs/user/_sources/run/help.txt
+            cpac run --help | sed -e "s/.*/   &/" >> docs/user/_sources/run/help.txt
+            mkdir -p docs/user/_sources/utils
+            printf "Usage: cpac utils\n\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\n.. code-block:: console\n\n   $ cpac utils --help\n\n" > docs/user/_sources/utils/help.txt
+            cpac utils --help | sed -e "s/.*/   &/" >> docs/user/_sources/utils/help.txt
+            deactivate
+      - run:
+          name: Build docs
+          command: ./bin/build
       - run:
           name: Configure git user
           command: |

CONTRIBUTING.md

@@ -46,6 +46,15 @@ where `{width}` and `{height}` are the values already present in the existing `w
 ```
 
 ## Environment notes
-* :snake: Until [C-PAC Python 2 → 3 conversion](https://github.com/FCP-INDI/C-PAC/issues/1114) is complete, these docs require Python 2.7 to build.
+* Because [C-PAC](https://github.com/FCP-INDI/C-PAC.git) and [cpac](https://github.com/FCP-INDI/cpac.git) have conflicting commandline commands, we first run any `cpac` commands in a virtual environment and spoof the `command-output` directive with `code-block` like 
+   ```RST
+   .. code-block:: console
+
+       cpac run --help
+       
+   .. program-output:: cpac_py run --help
+      :shell:
+      :ellipsis: 0,9
+   ```
 * :heavy_plus_sign: Check [`.circleci/config.yml`](https://github.com/FCP-INDI/fcp-indi.github.com/blob/source/.circleci/config.yml) of the branch you're working from for build dependencies.
 * :octocat: Set an environment variable `GITHUBTOKEN` to a [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) to increase [your API rate limit](https://developer.github.com/v3/#rate-limiting) from 60 to 5000 requests per hour (for getting [release notes from GitHub](https://github.com/FCP-INDI/C-PAC/releases)).

docs/user/_sources/appendix.txt

@@ -11,9 +11,7 @@ Before You Start
 
 C-PAC can be run without installing through any of these three options:
 
-* With a Docker container
-* On the Amazon AWS Cloud
-* Through OpenNeuro
+.. include:: run/without_installing.txt
 
 For more details, skip ahead to the :doc:`C-PAC Quickstart </quick>` or :doc:`Run C-PAC </running>` pages.
 
@@ -42,7 +40,7 @@ A Note About the \*nix Command Line
 
 Overview
 --------
-If you are using Ubuntu (version 12/14) you can download  and run `cpac_install.sh <https://github.com/FCP-INDI/C-PAC/blob/master/scripts/cpac_install.sh?raw=true>`__ with the following terminal commands as root:
+If you are using Ubuntu (version 12/14) you can download and run `cpac_install.sh <https://github.com/FCP-INDI/C-PAC/blob/master/scripts/cpac_install.sh?raw=true>`__ with the following terminal commands as root:
 
 .. code-block:: console
 
@@ -53,7 +51,9 @@ If you are using Ubuntu (version 12/14) you can download  and run `cpac_install.
 
 This will install all of the prerequisites and resources listed below. If you have previously installed versions of these, they will not be reinstalled, and the script will not check the release version to match the requirements of C-PAC.
 
-**Note**: This script is intended for Ubuntu Linux 12/14,  but it may work for older releases of Ubuntu.
+.. note::
+
+   This script is intended for Ubuntu Linux 12/14,  but it may work for older releases of Ubuntu.
 
 **For other operating systems**, this page will guide you through the following C-PAC installation steps:
 
@@ -129,9 +129,15 @@ OS-Specific Instructions
 
 Installing Python Dependencies
 """"""""""""""""""""""""""""""
-Please ensure that you are using Python 2.7 and above, but not Python 3 (since it is not backwards-compatible). Though many computers come with Python pre-installed, C-PAC relies on a number of special-purpose packages, which are listed below. Packages with an asterisk can be installed through `easy_install <https://pythonhosted.org/setuptools/easy_install.html>`__ or pip.  Installing `Anaconda <https://store.continuum.io/cshop/anaconda/>`__ (**64-bit version only**), `Miniconda <http://conda.pydata.org/miniconda.html>`__ or  `Enthought Canopy <https://www.enthought.com/products/canopy/>`__ and using a package manager can simplify installation greatly.  Instructions for Miniconda are given below.
+Please ensure that you are using Python 3.6 and above. Though many computers come with Python pre-installed, C-PAC relies on a number of special-purpose packages, which are listed below. Packages with an asterisk can be installed through `easy_install <https://pythonhosted.org/setuptools/easy_install.html>`__ or pip.  Installing `Anaconda <https://store.continuum.io/cshop/anaconda/>`__ (**64-bit version only**), `Miniconda <http://conda.pydata.org/miniconda.html>`__ or  `Enthought Canopy <https://www.enthought.com/products/canopy/>`__ and using a package manager can simplify installation greatly.  Instructions for Miniconda are given below.
+
+.. note::
+
+   :doc:`cpac (the simple Python commandline interface) <cpac>` and a local installation of C-PAC will conflict if installed in the same environment because both use the same commandline command (``cpac``).
 
-Note: Specific maximum versions are required for the following dependencies. C-PAC is currently (temporarily) not compatible with versions past the following:
+.. note::
+
+   Specific maximum versions are required for the following dependencies. C-PAC is currently (temporarily) not compatible with versions past the following:
 
 * `Nipype <http://nipype.readthedocs.io/en/latest/>`__ - version 1.1.2
 * `NetworkX <http://networkx.lanl.gov/>`__ - version 1.11
@@ -213,9 +219,13 @@ For AFNI to be available globally on your machine, you should then add the follo
 
 If you open a new shell and type `afni` the AFNI console should now appear.  If not, double-check the lines added to `/etc/bash.bashrc` for typos and make sure that `/opt/afni` contains the AFNI commands.
 
-**Note:** Regarding the Neurodebian repository: We have encountered compatibility issues in the past with the Neurodebian binary for AFNI.  For this reason, it is suggested that you follow the installation instructions above or the instructions from the AFNI homepage.
+.. note::
+
+   Regarding the Neurodebian repository: We have encountered compatibility issues in the past with the Neurodebian binary for AFNI.  For this reason, it is suggested that you follow the installation instructions above or the instructions from the AFNI homepage.
 
-**Note:** On some Ubuntu systems, AFNI can have trouble locating the libgsl.so.0 software library required for some of their tools (3dSkullStrip, and 3dHist). If you experience an error trying to run any of these tools, first attempt to re-install AFNI via the instructions on the AFNI homepage. If this does not resolve the issue, another solution involves locating your system's GSL library and re-configuring:
+.. note::
+
+   On some Ubuntu systems, AFNI can have trouble locating the libgsl.so.0 software library required for some of their tools (3dSkullStrip, and 3dHist). If you experience an error trying to run any of these tools, first attempt to re-install AFNI via the instructions on the AFNI homepage. If this does not resolve the issue, another solution involves locating your system's GSL library and re-configuring:
 
 .. code-block:: console
 
@@ -416,13 +426,17 @@ In a new terminal window, open iPython (or Python) and enter the command ``impor
 
 Once you are able to successfully ``import CPAC`` it is safe to delete any setup files downloaded during the install process (e.g. Nipype and C-PAC downloads, FSL install scripts, etc.), as they are no longer needed.
 
-**Note:** The test process described here only acts to confirm that the C-PAC python package has been correctly installed. To fully test C-PAC on your system, please see the :doc:`Benchmark Page </benchmark>`.
+.. note::
+
+   The test process described here only acts to confirm that the C-PAC python package (the core package, not :doc:`the simple Python commandline interface with the same name <cpac>`) has been correctly installed. To fully test C-PAC on your system, please see the :doc:`Benchmark Page </benchmark>`.
 
 Updating C-PAC
 """"""""""""""
 C-PAC is being actively developed, and new versions (containing bug fixes and new features) are often released. To update to the latest version, simply download it from the `C-PAC Homepage <http://fcp-indi.github.com/>`__ and repeat the instructions in the `Installing C-PAC` section above. A list of previous versions and the changes they contain is available on the :doc:`Release Notes Page </rnotes>`.
 
-**Note: If you are using Anaconda/Miniconda you may also use the following command (replacing 'cpac' with your environment name) to remove an old environment before creating a new environment to replace it.**
+.. note::
+
+   If you are using Anaconda/Miniconda you may also use the following command (replacing 'cpac' with your environment name) to remove an old environment before creating a new environment to replace it.**
 
  .. code-block:: console
 
@@ -439,4 +453,8 @@ Some network centrality features will not be available without compiling the C-b
 
     python setup.py build_ext --inplace
 
+.. note::
+
+   Unfortunately, it is not possible at this time to use the C-PAC GUI without installing C-PAC.
+
 .. include:: benchmark.txt

docs/user/_sources/cpac.txt

@@ -0,0 +1,87 @@
+cpac (Python package)
+---------------------
+
+`cpac <https://pypi.org/project/cpac/>`_ is available so that you can easily run analyses without needing interact with the container platform that allows you to run C-PAC without installing all of the underlying software.
+
+Currently the cpac supports Singularity (version ≥ 2.5 ≤ 3.0) and Docker.
+
+cpac requires Python 3.6 or greater. To get cpac, simply
+
+.. code-block:: console
+
+    pip install cpac
+
+As a quick example, in order to run C-PAC in participant mode, for one participant, using a BIDS dataset stored on your machine or server, and using the container image's default pipeline configuration:
+
+.. code-block:: console
+
+    cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant
+
+By default, the cpac will try Docker first and fall back to Singularity if Docker fails. If both fail, an exception is raised.
+
+You can specify a platform with the ``--platform docker`` or ``--platform singularity``. If you specify a platform without specifying an image, these are the defaults, using the first successfully found image:
+
+Usage
+`````
+.. include:: /cpac/help.txt
+
+``--platform docker``
+*********************
+
+* Look for ``fcpindi/c-pac:latest`` locally.
+* Pull ``fcpindi/c-pac:latest`` from Docker Hub.
+
+``--platform singularity``
+**************************
+
+* Look in the present working directory for any Singularity images. If more than one is found, use the most recently modified.
+* Pull ``FCP-INDI/C-PAC`` from Singularity Hub.
+* Pull ``fcpindi/c-pac:latest`` from Docker Hub and convert to a Singularity image.
+
+You can also specify a container image with an ``--image`` argument, passing an image name (e.g., ``fcpindi/c-pac``) for a Docker image or a filepath (e.g. ``~/singularity_images/C-PAC.sif``) for a Singularity image. You can also specify a ``--tag`` (e.g., ``latest`` or ``nightly``).
+
+You can also provide a link to an AWS S3 bucket containing a BIDS directory as the data source:
+
+.. code-block:: console
+
+    cpac run s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS /Users/You/some_folder_for_outputs participant
+
+In addition to the default pipeline, C-PAC comes packaged with a growing library of pre-configured pipelines that are ready to use. To run C-PAC with one of the pre-packaged pre-configured pipelines, simply invoke the ``--preconfig`` flag, shown below. See the full selection of pre-configured pipelines :doc:`here </preconfig>`.
+
+.. code-block:: console
+
+    cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs --preconfig anat-only
+
+To run C-PAC with a pipeline configuration file other than one of the pre-configured pipelines, assuming the configuration file is in the ``/Users/You/Documents`` directory:
+
+.. code-block:: console
+
+    cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant --pipeline_file /Users/You/Documents/pipeline_config.yml
+
+Finally, to run C-PAC with a specific data configuration file (instead of providing a BIDS data directory):
+
+.. code-block:: console
+
+    cpac run /Users/You/any_directory /Users/You/some_folder_for_outputs participant --data_config_file /Users/You/Documents/data_config.yml
+
+Note: we are still providing the postionally-required ``bids_dir`` input parameter. However C-PAC will not look for data in this directory when you provide a data configuration YAML with the ``--data_config_file`` flag. Providing ``.`` or ``$PWD`` will simply pass the present working directory. In addition, if the dataset in your data configuration file is not in BIDS format, just make sure to add the ``--skip_bids_validator`` flag at the end of your command to bypass the BIDS validation process.
+
+The full list of parameters and options that can be passed to C-PAC are shown below:
+
+.. include:: /run/help.txt
+
+.. include:: /utils/help.txt
+
+Note that any of the optional arguments above will over-ride any pipeline settings in the default pipeline or in the pipeline configuration file you provide via the ``--pipeline_file`` parameter.
+
+**Further usage notes:**
+
+* You can run only anatomical preprocessing easily, without modifying your data or pipeline configuration files, by providing the ``--anat_only`` flag.
+
+* As stated, the default behavior is to read data that is organized in the BIDS format. This includes data that is in Amazon AWS S3 by using the format ``s3://<bucket_name>/<bids_dir>`` for the ``bids_dir`` command line argument. Outputs can be written to S3 using the same format for the ``output_dir``. Credentials for accessing these buckets can be specified on the command line (using ``--aws_input_creds`` or ``--aws_output_creds``).
+
+* When the app is run, a data configuration file is written to the working directory. This directory can be specified with ``--working_dir`` or the directory from which you run ``cpac`` will be used. This file can be passed into subsequent runs, which avoids the overhead of re-parsing the BIDS input directory on each run (i.e. for cluster or cloud runs). These files can be generated without executing the C-PAC pipeline using the ``test_run`` command line argument.
+
+* The ``participant_label`` and ``participant_ndx`` arguments allow the user to specify which of the many datasets should be processed, which is useful when parallelizing the run of multiple participants.
+
+* If you want to pass runtime options to your container plaform (Docker or Singularity), you can pass them with ``-o`` or ``--container_options``.

docs/user/_sources/docker.txt

@@ -89,7 +89,4 @@ Note that any of the optional arguments above will over-ride any pipeline settin
 
 * The ``participant_label`` and ``participant_ndx`` arguments allow the user to specify which of the many datasets should be processed, which is useful when parallelizing the run of multiple participants.
 
-.. toctree::
-   :hidden:
 
-   /run/help.txt

docs/user/_sources/quick.txt

@@ -3,9 +3,9 @@ C-PAC Quickstart
 
 .. include:: quick_overview.txt
 
-.. include:: docker.txt
+.. include:: cpac.txt
 
-.. include:: singularity.txt
+For instructions to run in Docker or Singularity without installing cpac (Python package), see :ref:`All Run Options <running>`
 
 Default Pipeline
 ----------------
@@ -54,10 +54,10 @@ We currently have a publication in preparation, in the meantime please cite our
 
 	@ARTICLE{cpac2013,
 	    AUTHOR={Craddock, Cameron  and  Sikka, Sharad  and  Cheung, Brian  and  Khanuja, Ranjeet  and  Ghosh, Satrajit S
-	        and Yan, Chaogan  and  Li, Qingyang  and  Lurie, Daniel  and  Vogelstein, Joshua  and  Burns, Randal  and  
+	        and Yan, Chaogan  and  Li, Qingyang  and  Lurie, Daniel  and  Vogelstein, Joshua  and  Burns, Randal  and
 	        Colcombe, Stanley  and  Mennes, Maarten  and  Kelly, Clare  and  Di Martino, Adriana  and  Castellanos,
-	        Francisco Xavier  and  Milham, Michael},  
-	    TITLE={Towards Automated Analysis of Connectomes: The Configurable Pipeline for the Analysis of Connectomes (C-PAC)},     
+	        Francisco Xavier  and  Milham, Michael},
+	    TITLE={Towards Automated Analysis of Connectomes: The Configurable Pipeline for the Analysis of Connectomes (C-PAC)},
 	    JOURNAL={Frontiers in Neuroinformatics},
 	    YEAR={2013},
 	    NUMBER={42},

docs/user/_sources/quick_overview.txt

@@ -1,7 +1,7 @@
 Let's Go!
 ---------
 
-#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity.
+#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity. Using these containers can be facilitated by :doc:`cpac`.
 
 #. Containers are completely pre-built environments that help ensure reproducibility (same exact library versions, behavior, etc.).
 

docs/user/_sources/run/without_installing.txt

@@ -0,0 +1,3 @@
+* With a Docker or Singularity container (optionally with :doc:`a simple Python commandline interface <cpac>`)
+* On the Amazon AWS Cloud
+* Through OpenNeuro