From 977bb65e6d422d667b0f49132335b1f35e465614 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Fri, 8 May 2020 17:07:08 -0400 Subject: [PATCH 1/6] :books: Add C-PAC Python Package guide --- docs/user/_sources/cpac_py.txt | 86 +++++++++++++++++++++++++++ docs/user/_sources/docker.txt | 22 ------- docs/user/_sources/quick.txt | 8 ++- docs/user/_sources/quick_overview.txt | 2 +- docs/user/_sources/run/help.txt | 6 +- 5 files changed, 95 insertions(+), 29 deletions(-) create mode 100644 docs/user/_sources/cpac_py.txt diff --git a/docs/user/_sources/cpac_py.txt b/docs/user/_sources/cpac_py.txt new file mode 100644 index 000000000..d419cdffb --- /dev/null +++ b/docs/user/_sources/cpac_py.txt @@ -0,0 +1,86 @@ +cpac Python Package +------------------- + +A `C-PAC Python package `_ is available so that you can easily run analyses without needing interact with the container platform that allows you to run C-PAC without installing all of the underlying software. + +Currently the C-PAC Python package supports Singularity (version ≥ 2.5 ≤ 3.0) and Docker. + +The C-PAC Python package requires Python 3.6 or greater. To get the C-PAC Python package, simply + +.. code-block:: console + + pip install cpac-py + +As a quick example, in order to run C-PAC in participant mode, for one participant, using a BIDS dataset stored on your machine or server, and using the container image's default pipeline configuration: + +.. code-block:: console + + cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant + +By default, the C-PAC Python package will try Docker first and fall back to Singularity if Docker fails. If both fail, an exception is raised. + +You can specify a platform with the ``--platform docker`` or ``--platform singularity``. If you specify a platform without specifying an image, these are the defaults, using the first successfully found image: + +Docker +****** + +* Look for ``fcpindi/c-pac:latest`` locally. +* Pull ``fcpindi/c-pac:latest`` from Docker Hub. + +Singularity +*********** + +* Look in the present working directory for any Singularity images. If more than one is found, use the most recently modified. +* Pull ``FCP-INDI/C-PAC`` from Singularity Hub. +* Pull ``fcpindi/c-pac:latest`` from Docker Hub and convert to a Singularity image. + +You can also specify a container image with an ``--image`` argument, passing an image name (e.g., ``fcpindi/c-pac``) for a Docker image or a filepath (e.g. ``~/singularity_images/C-PAC.sif``) for a Singularity image. You can also specify a ``--tag`` (e.g., ``latest`` or ``nightly``). + +You can also provide a link to an AWS S3 bucket containing a BIDS directory as the data source: + +.. code-block:: console + + cpac run s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS /Users/You/some_folder_for_outputs participant + +In addition to the default pipeline, C-PAC comes packaged with a growing library of pre-configured pipelines that are ready to use. To run C-PAC with one of the pre-packaged pre-configured pipelines, simply invoke the ``--preconfig`` flag, shown below. See the full selection of pre-configured pipelines :doc:`here `. + +.. code-block:: console + + cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs --preconfig anat-only + +To run C-PAC with a pipeline configuration file other than one of the pre-configured pipelines, assuming the configuration file is in the ``/Users/You/Documents`` directory: + +.. code-block:: console + + cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant --pipeline_file /Users/You/Documents/pipeline_config.yml + +Finally, to run C-PAC with a specific data configuration file (instead of providing a BIDS data directory): + +.. code-block:: console + + cpac run /Users/You/any_directory /Users/You/some_folder_for_outputs participant --data_config_file /Users/You/Documents/data_config.yml + +Note: we are still providing the postionally-required ``bids_dir`` input parameter. However C-PAC will not look for data in this directory when you provide a data configuration YAML with the ``--data_config_file`` flag. Providing ``.`` or ``$PWD`` will simply pass the present working directory. In addition, if the dataset in your data configuration file is not in BIDS format, just make sure to add the ``--skip_bids_validator`` flag at the end of your command to bypass the BIDS validation process. + +The full list of parameters and options that can be passed to the Docker container are shown below: + +.. include:: /run/help.txt + +Note that any of the optional arguments above will over-ride any pipeline settings in the default pipeline or in the pipeline configuration file you provide via the ``--pipeline_file`` parameter. + +**Further usage notes:** + +* You can run only anatomical preprocessing easily, without modifying your data or pipeline configuration files, by providing the ``--anat_only`` flag. + +* As stated, the default behavior is to read data that is organized in the BIDS format. This includes data that is in Amazon AWS S3 by using the format ``s3:///`` for the ``bids_dir`` command line argument. Outputs can be written to S3 using the same format for the ``output_dir``. Credentials for accessing these buckets can be specified on the command line (using ``--aws_input_creds`` or ``--aws_output_creds``). + +* When the app is run, a data configuration file is written to the working directory. This directory can be specified with ``--working_dir`` or the directory from which you run ``cpac`` will be used. This file can be passed into subsequent runs, which avoids the overhead of re-parsing the BIDS input directory on each run (i.e. for cluster or cloud runs). These files can be generated without executing the C-PAC pipeline using the ``test_run`` command line argument. + +* The ``participant_label`` and ``participant_ndx`` arguments allow the user to specify which of the many datasets should be processed, which is useful when parallelizing the run of multiple participants. + +* If you want to pass runtime options to your container plaform (Docker or Singularity), you can pass them with ``-o`` or ``--container_options``. + +.. toctree:: + :hidden: + + /run/help.txt diff --git a/docs/user/_sources/docker.txt b/docs/user/_sources/docker.txt index 00362d37d..2e258a599 100644 --- a/docs/user/_sources/docker.txt +++ b/docs/user/_sources/docker.txt @@ -73,25 +73,3 @@ Finally, to run the Docker container with a specific data configuration file (in Note: we are still providing ``/bids_dataset`` to the ``bids_dir`` input parameter. However, we have mapped this to any directory on your machine, as C-PAC will not look for data in this directory when you provide a data configuration YAML with the ``--data_config_file`` flag. In addition, if the dataset in your data configuration file is not in BIDS format, just make sure to add the ``--skip_bids_validator`` flag at the end of your command to bypass the BIDS validation process. -The full list of parameters and options that can be passed to the Docker container are shown below: - -.. include:: /run/help.txt - -Note that any of the optional arguments above will over-ride any pipeline settings in the default pipeline or in the pipeline configuration file you provide via the ``--pipeline_file`` parameter. - -**Further usage notes:** - -* You can run only anatomical preprocessing easily, without modifying your data or pipeline configuration files, by providing the ``--anat_only`` flag. - -* A GUI can be invoked to assist in pipeline customization by specifying the ``GUI`` command line argument, as opposed to ``participant`` (this currently only works for Singularity containers). - -* As stated, the default behavior is to read data that is organized in the BIDS format. This includes data that is in Amazon AWS S3 by using the format ``s3:///`` for the ``bids_dir`` command line argument. Outputs can be written to S3 using the same format for the ``output_dir``. Credentials for accessing these buckets can be specified on the command line (using ``--aws_input_creds`` or ``--aws_output_creds``). - -* When the app is run, a data configuration file is written to the working directory. This file can be passed into subsequent runs, which avoids the overhead of re-parsing the BIDS input directory on each run (i.e. for cluster or cloud runs). These files can be generated without executing the C-PAC pipeline using the test_run command line argument. - -* The ``participant_label`` and ``participant_ndx`` arguments allow the user to specify which of the many datasets should be processed, which is useful when parallelizing the run of multiple participants. - -.. toctree:: - :hidden: - - /run/help.txt diff --git a/docs/user/_sources/quick.txt b/docs/user/_sources/quick.txt index 7a0d8b0c8..2d38db96f 100644 --- a/docs/user/_sources/quick.txt +++ b/docs/user/_sources/quick.txt @@ -3,6 +3,8 @@ C-PAC Quickstart .. include:: quick_overview.txt +.. include:: cpac_py.txt + .. include:: docker.txt .. include:: singularity.txt @@ -54,10 +56,10 @@ We currently have a publication in preparation, in the meantime please cite our @ARTICLE{cpac2013, AUTHOR={Craddock, Cameron and Sikka, Sharad and Cheung, Brian and Khanuja, Ranjeet and Ghosh, Satrajit S - and Yan, Chaogan and Li, Qingyang and Lurie, Daniel and Vogelstein, Joshua and Burns, Randal and + and Yan, Chaogan and Li, Qingyang and Lurie, Daniel and Vogelstein, Joshua and Burns, Randal and Colcombe, Stanley and Mennes, Maarten and Kelly, Clare and Di Martino, Adriana and Castellanos, - Francisco Xavier and Milham, Michael}, - TITLE={Towards Automated Analysis of Connectomes: The Configurable Pipeline for the Analysis of Connectomes (C-PAC)}, + Francisco Xavier and Milham, Michael}, + TITLE={Towards Automated Analysis of Connectomes: The Configurable Pipeline for the Analysis of Connectomes (C-PAC)}, JOURNAL={Frontiers in Neuroinformatics}, YEAR={2013}, NUMBER={42}, diff --git a/docs/user/_sources/quick_overview.txt b/docs/user/_sources/quick_overview.txt index 269fa0006..8de6563e6 100644 --- a/docs/user/_sources/quick_overview.txt +++ b/docs/user/_sources/quick_overview.txt @@ -1,7 +1,7 @@ Let's Go! --------- -#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity. +#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity. Using these containers can be facilitated by the C-PAC Python package. #. Containers are completely pre-built environments that help ensure reproducibility (same exact library versions, behavior, etc.). diff --git a/docs/user/_sources/run/help.txt b/docs/user/_sources/run/help.txt index 59db90661..bb6e4ebd6 100644 --- a/docs/user/_sources/run/help.txt +++ b/docs/user/_sources/run/help.txt @@ -1,4 +1,4 @@ -cpac run --help -``````````````` +Usage +````` -.. program-output:: python -m /build/C-PAC/dev/docker_data/run --help +.. program-output:: cpac run --help | tail --lines=+10 From 3e22beed6eb15006e4ae0834235b746818eb4006 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Fri, 8 May 2020 17:41:03 -0400 Subject: [PATCH 2/6] :books: Get `run --help` from cpac-py --- .circleci/config.yml | 1 + docs/user/_sources/run/help.txt | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 69dd9687a..06eeaf271 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -15,6 +15,7 @@ jobs: apt-get update && apt-get -y install git rsync python-dev graphviz libgraphviz-dev pkg-config python3-sphinx pip install --user -r https://raw.githubusercontent.com/FCP-INDI/C-PAC/develop/requirements.txt pip install --user git+https://github.com/FCP-INDI/C-PAC.git sphinx m2r numpydoc PyGithub sphinxcontrib-programoutput + pip install cpac-py git clone https://github.com/FCP-INDI/C-PAC.git - run: name: Build docs diff --git a/docs/user/_sources/run/help.txt b/docs/user/_sources/run/help.txt index bb6e4ebd6..2f9479652 100644 --- a/docs/user/_sources/run/help.txt +++ b/docs/user/_sources/run/help.txt @@ -1,4 +1,4 @@ -Usage -````` +cpac run --help +``````````````` .. program-output:: cpac run --help | tail --lines=+10 From a04f41b33078041f368889a6d44c809b1fb84e7b Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Sat, 9 May 2020 09:33:16 -0400 Subject: [PATCH 3/6] :green_heart: setup_remote_docker --- .circleci/config.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 06eeaf271..7985954f7 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -17,6 +17,7 @@ jobs: pip install --user git+https://github.com/FCP-INDI/C-PAC.git sphinx m2r numpydoc PyGithub sphinxcontrib-programoutput pip install cpac-py git clone https://github.com/FCP-INDI/C-PAC.git + - setup_remote_docker - run: name: Build docs command: | From 2bf027379f6431588f7e260cab0d31a0119f7d82 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Mon, 11 May 2020 12:19:51 -0400 Subject: [PATCH 4/6] :books: Usage --- docs/user/_sources/cpac_py.txt | 17 ++++++++++++----- docs/user/_sources/run/help.txt | 6 ++++-- docs/user/_sources/utils/help.txt | 6 ++++++ 3 files changed, 22 insertions(+), 7 deletions(-) create mode 100644 docs/user/_sources/utils/help.txt diff --git a/docs/user/_sources/cpac_py.txt b/docs/user/_sources/cpac_py.txt index d419cdffb..9e4780c8c 100644 --- a/docs/user/_sources/cpac_py.txt +++ b/docs/user/_sources/cpac_py.txt @@ -21,14 +21,18 @@ By default, the C-PAC Python package will try Docker first and fall back to Sing You can specify a platform with the ``--platform docker`` or ``--platform singularity``. If you specify a platform without specifying an image, these are the defaults, using the first successfully found image: -Docker -****** +Usage +````` +.. command-output:: cpac --help + +--platform docker +***************** * Look for ``fcpindi/c-pac:latest`` locally. * Pull ``fcpindi/c-pac:latest`` from Docker Hub. -Singularity -*********** +--platform singularity +********************** * Look in the present working directory for any Singularity images. If more than one is found, use the most recently modified. * Pull ``FCP-INDI/C-PAC`` from Singularity Hub. @@ -62,10 +66,12 @@ Finally, to run C-PAC with a specific data configuration file (instead of provid Note: we are still providing the postionally-required ``bids_dir`` input parameter. However C-PAC will not look for data in this directory when you provide a data configuration YAML with the ``--data_config_file`` flag. Providing ``.`` or ``$PWD`` will simply pass the present working directory. In addition, if the dataset in your data configuration file is not in BIDS format, just make sure to add the ``--skip_bids_validator`` flag at the end of your command to bypass the BIDS validation process. -The full list of parameters and options that can be passed to the Docker container are shown below: +The full list of parameters and options that can be passed to C-PAC are shown below: .. include:: /run/help.txt +.. include:: /utils/help.txt + Note that any of the optional arguments above will over-ride any pipeline settings in the default pipeline or in the pipeline configuration file you provide via the ``--pipeline_file`` parameter. **Further usage notes:** @@ -84,3 +90,4 @@ Note that any of the optional arguments above will over-ride any pipeline settin :hidden: /run/help.txt + /utils/help.txt diff --git a/docs/user/_sources/run/help.txt b/docs/user/_sources/run/help.txt index 2f9479652..f70fb5425 100644 --- a/docs/user/_sources/run/help.txt +++ b/docs/user/_sources/run/help.txt @@ -1,4 +1,6 @@ -cpac run --help +Usage: cpac run ``````````````` -.. program-output:: cpac run --help | tail --lines=+10 +.. command-output:: cpac run --help + :shell: + :ellipsis: 0,9 \ No newline at end of file diff --git a/docs/user/_sources/utils/help.txt b/docs/user/_sources/utils/help.txt new file mode 100644 index 000000000..976b3c886 --- /dev/null +++ b/docs/user/_sources/utils/help.txt @@ -0,0 +1,6 @@ +Usage: cpac utils +````````````````` + +.. command-output:: cpac utils --help + :shell: + :ellipsis: 0,9 \ No newline at end of file From 06b3437f18dc2de24eec144a4758bada7dbb5ac5 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Mon, 11 May 2020 12:46:00 -0400 Subject: [PATCH 5/6] :construction: :books: cpac_py Documentation --- docs/user/_sources/cpac_py.txt | 8 ++++---- docs/user/_sources/pipeline_config.txt | 2 +- docs/user/_sources/quick.txt | 4 +--- docs/user/_sources/running.txt | 12 +++++++++--- docs/user/_sources/subject_list_config.txt | 14 +++++++++++++- 5 files changed, 28 insertions(+), 12 deletions(-) diff --git a/docs/user/_sources/cpac_py.txt b/docs/user/_sources/cpac_py.txt index 9e4780c8c..c2323be99 100644 --- a/docs/user/_sources/cpac_py.txt +++ b/docs/user/_sources/cpac_py.txt @@ -25,14 +25,14 @@ Usage ````` .. command-output:: cpac --help ---platform docker -***************** +``--platform docker`` +********************* * Look for ``fcpindi/c-pac:latest`` locally. * Pull ``fcpindi/c-pac:latest`` from Docker Hub. ---platform singularity -********************** +``--platform singularity`` +************************** * Look in the present working directory for any Singularity images. If more than one is found, use the most recently modified. * Pull ``FCP-INDI/C-PAC`` from Singularity Hub. diff --git a/docs/user/_sources/pipeline_config.txt b/docs/user/_sources/pipeline_config.txt index 384a64fc1..0d78689e8 100644 --- a/docs/user/_sources/pipeline_config.txt +++ b/docs/user/_sources/pipeline_config.txt @@ -25,7 +25,7 @@ Definitions .. include:: /glossary/derivative.txt :start-line: 3 -..include:: design_a_pipeline.txt +.. include:: design_a_pipeline.txt Using a Text Editor ------------------- diff --git a/docs/user/_sources/quick.txt b/docs/user/_sources/quick.txt index 2d38db96f..16c7a6697 100644 --- a/docs/user/_sources/quick.txt +++ b/docs/user/_sources/quick.txt @@ -5,9 +5,7 @@ C-PAC Quickstart .. include:: cpac_py.txt -.. include:: docker.txt - -.. include:: singularity.txt +For instructions to run in Docker or Singularity without installing the C-PAC Python package, see :ref:`All Run Options ` Default Pipeline ---------------- diff --git a/docs/user/_sources/running.txt b/docs/user/_sources/running.txt index 44a7ba20b..3f90b8612 100644 --- a/docs/user/_sources/running.txt +++ b/docs/user/_sources/running.txt @@ -22,6 +22,10 @@ If you re-run C-PAC with an output directory containing a working directory (fro C-PAC migrated from Python 2 to Python 3 in v1.6.2 (see `release notes `_). If your working directory contains Python 2 pickles from an older version of C-PAC and you want to continue to use this working directory, run:: + cpac utils repickle /path/to/working_dir + + or:: + docker run -i --rm --user $(id -u):$(id -g) -v /path/to/working_dir:/working fcpindi/c-pac:latest /bids_dir /outputs cli -- utils repickle /working or:: @@ -34,12 +38,14 @@ If you re-run C-PAC with an output directory containing a working directory (fro :ref:`Common Issue: I'm re-running a pipeline, but I am receiving many crashes ` +.. include:: cpac_py.txt + .. include:: docker.txt .. include:: singularity.txt On the AWS Cloud ------------------------------- +---------------- The C-PAC team has released an Amazon Marketplace AMI, making it easier for researchers to use C-PAC in the cloud. You can use the AMI to either launch a single machine for basic runs or create a high performance computing (HPC) cluster using Starcluster. Clusters can be dynamically scaled up as your computational needs increase. Detailed explanations of cloud computing and HPC are beyond the scope of this documentation, but we will define a few key terms before we start. If these terms are familiar, you may skip them and proceed to later sections. @@ -62,7 +68,7 @@ The C-PAC team has released an Amazon Marketplace AMI, making it easier for rese Lastly, it would be important to review any terms related to :doc:`the Sun Grid Engine job scheduler `. Creating AWS Access and Network Keys -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Before you can create a single C-PAC machine or a C-PAC HPC cluster, you must first generate credentials that will allow you to log into any AWS instance that you create. The following steps will walk you through the process of creating all the necessary credentials and encryption keys that you will need. @@ -85,7 +91,7 @@ Before you can create a single C-PAC machine or a C-PAC HPC cluster, you must fi #. On your local drive, open a terminal and run the following command: ``chmod 400 /path/to/pem/file`` Starting a Single C-PAC Instance via the AWS Console -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Now that you have generated the access keys and a pem file, you may launch a single instance via Amazon's web interface by following the steps below. If you are planning on processing many subjects or obtaining computationally-intensive derivatives (such as network centrality), you should use Starcluster instead. diff --git a/docs/user/_sources/subject_list_config.txt b/docs/user/_sources/subject_list_config.txt index 9937951f9..9daf13834 100644 --- a/docs/user/_sources/subject_list_config.txt +++ b/docs/user/_sources/subject_list_config.txt @@ -25,13 +25,17 @@ You can configure the settings explained below in the **data settings** YAML fil If you don't already have a data settings YAML file, you can generate one in your current directory by running:: + cpac utils data_config new_settings_template + +or:: + singularity run C-PAC_latest.sif bids_dir outputs_dir cli -- utils data_config new_settings_template or:: docker run -i --rm --user $(id -u) -v /path/to/data_config:/scratch -w="/scratch" fcpindi/c-pac:latest bids_dir outputs_dir cli -- utils data_config new_settings_template -For either of the above commands, ``bids_dir`` and ``outputs_dir`` are required but not used, so any non-null string will work for those positional arguments. +If running a container directly (either of the latter options rather than the first option, which is through the :ref:`C-PAC Python package `), ``bids_dir`` and ``outputs_dir`` are required but not used, so any non-null string will work for those positional arguments. The Singularity command will create ``data_settings.yml`` in the current working directory you call the command from. @@ -119,6 +123,10 @@ You can also fill in the AWS credentials file field, and the inclusion and exclu Once your data settings file is ready, generate your data configuration file by running:: + cpac utils data_config build /path/to/data_settings.yml + +or:: + singularity run C-PAC_latest.sif bids_dir outputs_dir cli -- utils data_config build /path/to/data_settings.yml or:: @@ -151,6 +159,10 @@ You can also fill in the AWS credentials file field, and the inclusion and exclu Once your data settings file is ready, generate your data configuration file by running:: + cpac utils data_config build /path/to/data_settings.yml + +or:: + singularity run C-PAC_latest.sif bids_dir outputs_dir cli -- utils data_config build /path/to/data_settings.yml or:: From 98f768d0a8a7cb64d20f16cb5023996be0f87f3a Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Thu, 23 Jul 2020 11:34:52 -0400 Subject: [PATCH 6/6] =?UTF-8?q?:truck:=20cpac-py=20=E2=86=92=20cpac?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ref https://github.com/shnizzedy/cpac/issues/2 --- .circleci/config.yml | 24 +++++++++-- CONTRIBUTING.md | 11 ++++- docs/user/_sources/appendix.txt | 40 +++++++++++++------ docs/user/_sources/{cpac_py.txt => cpac.txt} | 24 +++++------ docs/user/_sources/quick.txt | 4 +- docs/user/_sources/quick_overview.txt | 2 +- docs/user/_sources/run/help.txt | 6 --- docs/user/_sources/run/without_installing.txt | 3 ++ docs/user/_sources/running.txt | 10 ++--- docs/user/_sources/subject_list_config.txt | 2 +- docs/user/_sources/utils/help.txt | 6 --- 11 files changed, 78 insertions(+), 54 deletions(-) rename docs/user/_sources/{cpac_py.txt => cpac.txt} (86%) delete mode 100644 docs/user/_sources/run/help.txt create mode 100644 docs/user/_sources/run/without_installing.txt delete mode 100644 docs/user/_sources/utils/help.txt diff --git a/.circleci/config.yml b/.circleci/config.yml index 7985954f7..f80868d79 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -15,14 +15,30 @@ jobs: apt-get update && apt-get -y install git rsync python-dev graphviz libgraphviz-dev pkg-config python3-sphinx pip install --user -r https://raw.githubusercontent.com/FCP-INDI/C-PAC/develop/requirements.txt pip install --user git+https://github.com/FCP-INDI/C-PAC.git sphinx m2r numpydoc PyGithub sphinxcontrib-programoutput - pip install cpac-py git clone https://github.com/FCP-INDI/C-PAC.git + python -m venv ~/simple + source ~/simple/bin/activate + pip install cpac + deactivate - setup_remote_docker - run: - name: Build docs + name: Run cpac commands command: | - ./bin/build - + source ~/simple/bin/activate + cpac run --help + mkdir -p docs/user/_sources/cpac + printf ".. code-block:: console\n\n $ cpac --help\n\n" > docs/user/_sources/cpac/help.txt + cpac --help | sed -e "s/.*/ &/" >> docs/user/_sources/cpac/help.txt + mkdir -p docs/user/_sources/run + printf "Usage: cpac run\n\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\n.. code-block:: console\n\n $ cpac run --help\n\n" > docs/user/_sources/run/help.txt + cpac run --help | sed -e "s/.*/ &/" >> docs/user/_sources/run/help.txt + mkdir -p docs/user/_sources/utils + printf "Usage: cpac utils\n\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\n.. code-block:: console\n\n $ cpac utils --help\n\n" > docs/user/_sources/utils/help.txt + cpac utils --help | sed -e "s/.*/ &/" >> docs/user/_sources/utils/help.txt + deactivate + - run: + name: Build docs + command: ./bin/build - run: name: Configure git user command: | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 27e2dd329..6a1c172d1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,6 +46,15 @@ where `{width}` and `{height}` are the values already present in the existing `w ``` ## Environment notes -* :snake: Until [C-PAC Python 2 → 3 conversion](https://github.com/FCP-INDI/C-PAC/issues/1114) is complete, these docs require Python 2.7 to build. +* Because [C-PAC](https://github.com/FCP-INDI/C-PAC.git) and [cpac](https://github.com/FCP-INDI/cpac.git) have conflicting commandline commands, we first run any `cpac` commands in a virtual environment and spoof the `command-output` directive with `code-block` like + ```RST + .. code-block:: console + + cpac run --help + + .. program-output:: cpac_py run --help + :shell: + :ellipsis: 0,9 + ``` * :heavy_plus_sign: Check [`.circleci/config.yml`](https://github.com/FCP-INDI/fcp-indi.github.com/blob/source/.circleci/config.yml) of the branch you're working from for build dependencies. * :octocat: Set an environment variable `GITHUBTOKEN` to a [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) to increase [your API rate limit](https://developer.github.com/v3/#rate-limiting) from 60 to 5000 requests per hour (for getting [release notes from GitHub](https://github.com/FCP-INDI/C-PAC/releases)). diff --git a/docs/user/_sources/appendix.txt b/docs/user/_sources/appendix.txt index 5be926c9e..072efc749 100644 --- a/docs/user/_sources/appendix.txt +++ b/docs/user/_sources/appendix.txt @@ -11,9 +11,7 @@ Before You Start C-PAC can be run without installing through any of these three options: -* With a Docker container -* On the Amazon AWS Cloud -* Through OpenNeuro +.. include:: run/without_installing.txt For more details, skip ahead to the :doc:`C-PAC Quickstart ` or :doc:`Run C-PAC ` pages. @@ -42,7 +40,7 @@ A Note About the \*nix Command Line Overview -------- -If you are using Ubuntu (version 12/14) you can download and run `cpac_install.sh `__ with the following terminal commands as root: +If you are using Ubuntu (version 12/14) you can download and run `cpac_install.sh `__ with the following terminal commands as root: .. code-block:: console @@ -53,7 +51,9 @@ If you are using Ubuntu (version 12/14) you can download and run `cpac_install. This will install all of the prerequisites and resources listed below. If you have previously installed versions of these, they will not be reinstalled, and the script will not check the release version to match the requirements of C-PAC. -**Note**: This script is intended for Ubuntu Linux 12/14, but it may work for older releases of Ubuntu. +.. note:: + + This script is intended for Ubuntu Linux 12/14, but it may work for older releases of Ubuntu. **For other operating systems**, this page will guide you through the following C-PAC installation steps: @@ -129,9 +129,15 @@ OS-Specific Instructions Installing Python Dependencies """""""""""""""""""""""""""""" -Please ensure that you are using Python 2.7 and above, but not Python 3 (since it is not backwards-compatible). Though many computers come with Python pre-installed, C-PAC relies on a number of special-purpose packages, which are listed below. Packages with an asterisk can be installed through `easy_install `__ or pip. Installing `Anaconda `__ (**64-bit version only**), `Miniconda `__ or `Enthought Canopy `__ and using a package manager can simplify installation greatly. Instructions for Miniconda are given below. +Please ensure that you are using Python 3.6 and above. Though many computers come with Python pre-installed, C-PAC relies on a number of special-purpose packages, which are listed below. Packages with an asterisk can be installed through `easy_install `__ or pip. Installing `Anaconda `__ (**64-bit version only**), `Miniconda `__ or `Enthought Canopy `__ and using a package manager can simplify installation greatly. Instructions for Miniconda are given below. + +.. note:: + + :doc:`cpac (the simple Python commandline interface) ` and a local installation of C-PAC will conflict if installed in the same environment because both use the same commandline command (``cpac``). -Note: Specific maximum versions are required for the following dependencies. C-PAC is currently (temporarily) not compatible with versions past the following: +.. note:: + + Specific maximum versions are required for the following dependencies. C-PAC is currently (temporarily) not compatible with versions past the following: * `Nipype `__ - version 1.1.2 * `NetworkX `__ - version 1.11 @@ -214,9 +220,13 @@ For AFNI to be available globally on your machine, you should then add the follo If you open a new shell and type `afni` the AFNI console should now appear. If not, double-check the lines added to `/etc/bash.bashrc` for typos and make sure that `/opt/afni` contains the AFNI commands. -**Note:** Regarding the Neurodebian repository: We have encountered compatibility issues in the past with the Neurodebian binary for AFNI. For this reason, it is suggested that you follow the installation instructions above or the instructions from the AFNI homepage. +.. note:: + + Regarding the Neurodebian repository: We have encountered compatibility issues in the past with the Neurodebian binary for AFNI. For this reason, it is suggested that you follow the installation instructions above or the instructions from the AFNI homepage. -**Note:** On some Ubuntu systems, AFNI can have trouble locating the libgsl.so.0 software library required for some of their tools (3dSkullStrip, and 3dHist). If you experience an error trying to run any of these tools, first attempt to re-install AFNI via the instructions on the AFNI homepage. If this does not resolve the issue, another solution involves locating your system's GSL library and re-configuring: +.. note:: + + On some Ubuntu systems, AFNI can have trouble locating the libgsl.so.0 software library required for some of their tools (3dSkullStrip, and 3dHist). If you experience an error trying to run any of these tools, first attempt to re-install AFNI via the instructions on the AFNI homepage. If this does not resolve the issue, another solution involves locating your system's GSL library and re-configuring: .. code-block:: console @@ -417,13 +427,17 @@ In a new terminal window, open iPython (or Python) and enter the command ``impor Once you are able to successfully ``import CPAC`` it is safe to delete any setup files downloaded during the install process (e.g. Nipype and C-PAC downloads, FSL install scripts, etc.), as they are no longer needed. -**Note:** The test process described here only acts to confirm that the C-PAC python package has been correctly installed. To fully test C-PAC on your system, please see the :doc:`Benchmark Page `. +.. note:: + + The test process described here only acts to confirm that the C-PAC python package (the core package, not :doc:`the simple Python commandline interface with the same name `) has been correctly installed. To fully test C-PAC on your system, please see the :doc:`Benchmark Page `. Updating C-PAC """""""""""""" C-PAC is being actively developed, and new versions (containing bug fixes and new features) are often released. To update to the latest version, simply download it from the `C-PAC Homepage `__ and repeat the instructions in the `Installing C-PAC` section above. A list of previous versions and the changes they contain is available on the :doc:`Release Notes Page `. -**Note: If you are using Anaconda/Miniconda you may also use the following command (replacing 'cpac' with your environment name) to remove an old environment before creating a new environment to replace it.** +.. note:: + + If you are using Anaconda/Miniconda you may also use the following command (replacing 'cpac' with your environment name) to remove an old environment before creating a new environment to replace it.** .. code-block:: console @@ -440,6 +454,8 @@ Some network centrality features will not be available without compiling the C-b python setup.py build_ext --inplace -**Note:** Unfortunately, it is not possible at this time to use the C-PAC GUI without installing C-PAC. +.. note:: + + Unfortunately, it is not possible at this time to use the C-PAC GUI without installing C-PAC. .. include:: benchmark.txt \ No newline at end of file diff --git a/docs/user/_sources/cpac_py.txt b/docs/user/_sources/cpac.txt similarity index 86% rename from docs/user/_sources/cpac_py.txt rename to docs/user/_sources/cpac.txt index c2323be99..9e429ec16 100644 --- a/docs/user/_sources/cpac_py.txt +++ b/docs/user/_sources/cpac.txt @@ -1,15 +1,15 @@ -cpac Python Package -------------------- +cpac (Python package) +--------------------- -A `C-PAC Python package `_ is available so that you can easily run analyses without needing interact with the container platform that allows you to run C-PAC without installing all of the underlying software. +`cpac `_ is available so that you can easily run analyses without needing interact with the container platform that allows you to run C-PAC without installing all of the underlying software. -Currently the C-PAC Python package supports Singularity (version ≥ 2.5 ≤ 3.0) and Docker. +Currently the cpac supports Singularity (version ≥ 2.5 ≤ 3.0) and Docker. -The C-PAC Python package requires Python 3.6 or greater. To get the C-PAC Python package, simply +cpac requires Python 3.6 or greater. To get cpac, simply .. code-block:: console - pip install cpac-py + pip install cpac As a quick example, in order to run C-PAC in participant mode, for one participant, using a BIDS dataset stored on your machine or server, and using the container image's default pipeline configuration: @@ -17,13 +17,13 @@ As a quick example, in order to run C-PAC in participant mode, for one participa cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant -By default, the C-PAC Python package will try Docker first and fall back to Singularity if Docker fails. If both fail, an exception is raised. +By default, the cpac will try Docker first and fall back to Singularity if Docker fails. If both fail, an exception is raised. You can specify a platform with the ``--platform docker`` or ``--platform singularity``. If you specify a platform without specifying an image, these are the defaults, using the first successfully found image: Usage ````` -.. command-output:: cpac --help +.. include:: /cpac/help.txt ``--platform docker`` ********************* @@ -84,10 +84,4 @@ Note that any of the optional arguments above will over-ride any pipeline settin * The ``participant_label`` and ``participant_ndx`` arguments allow the user to specify which of the many datasets should be processed, which is useful when parallelizing the run of multiple participants. -* If you want to pass runtime options to your container plaform (Docker or Singularity), you can pass them with ``-o`` or ``--container_options``. - -.. toctree:: - :hidden: - - /run/help.txt - /utils/help.txt +* If you want to pass runtime options to your container plaform (Docker or Singularity), you can pass them with ``-o`` or ``--container_options``. \ No newline at end of file diff --git a/docs/user/_sources/quick.txt b/docs/user/_sources/quick.txt index 16c7a6697..39956215f 100644 --- a/docs/user/_sources/quick.txt +++ b/docs/user/_sources/quick.txt @@ -3,9 +3,9 @@ C-PAC Quickstart .. include:: quick_overview.txt -.. include:: cpac_py.txt +.. include:: cpac.txt -For instructions to run in Docker or Singularity without installing the C-PAC Python package, see :ref:`All Run Options ` +For instructions to run in Docker or Singularity without installing cpac (Python package), see :ref:`All Run Options ` Default Pipeline ---------------- diff --git a/docs/user/_sources/quick_overview.txt b/docs/user/_sources/quick_overview.txt index 8de6563e6..12b80d62c 100644 --- a/docs/user/_sources/quick_overview.txt +++ b/docs/user/_sources/quick_overview.txt @@ -1,7 +1,7 @@ Let's Go! --------- -#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity. Using these containers can be facilitated by the C-PAC Python package. +#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity. Using these containers can be facilitated by :doc:`cpac`. #. Containers are completely pre-built environments that help ensure reproducibility (same exact library versions, behavior, etc.). diff --git a/docs/user/_sources/run/help.txt b/docs/user/_sources/run/help.txt deleted file mode 100644 index f70fb5425..000000000 --- a/docs/user/_sources/run/help.txt +++ /dev/null @@ -1,6 +0,0 @@ -Usage: cpac run -``````````````` - -.. command-output:: cpac run --help - :shell: - :ellipsis: 0,9 \ No newline at end of file diff --git a/docs/user/_sources/run/without_installing.txt b/docs/user/_sources/run/without_installing.txt new file mode 100644 index 000000000..8b9882352 --- /dev/null +++ b/docs/user/_sources/run/without_installing.txt @@ -0,0 +1,3 @@ +* With a Docker or Singularity container (optionally with :doc:`a simple Python commandline interface `) +* On the Amazon AWS Cloud +* Through OpenNeuro \ No newline at end of file diff --git a/docs/user/_sources/running.txt b/docs/user/_sources/running.txt index 3f90b8612..98022135a 100644 --- a/docs/user/_sources/running.txt +++ b/docs/user/_sources/running.txt @@ -5,14 +5,12 @@ Overview As with configuring the subject list, pipeline configuration, and group analysis files, there are two ways of executing a C-PAC run: -* Using C-PAC's command line interface -* Using the main dialog in the C-PAC GUI +* Using C-PAC's :doc:`simple command line interface ` +* Using C-PAC's command line interface in a container or direct installation In addition to running C-PAC traditionally on your own local computer or on a server, there are three other avenues through which you can run C-PAC without going through the install process: -* With a Docker container -* On the Amazon AWS Cloud -* Through OpenNeuro +.. include:: run/without_installing.txt More details of these options are available below. @@ -38,7 +36,7 @@ If you re-run C-PAC with an output directory containing a working directory (fro :ref:`Common Issue: I'm re-running a pipeline, but I am receiving many crashes ` -.. include:: cpac_py.txt +.. include:: cpac.txt .. include:: docker.txt diff --git a/docs/user/_sources/subject_list_config.txt b/docs/user/_sources/subject_list_config.txt index 9daf13834..375f411a8 100644 --- a/docs/user/_sources/subject_list_config.txt +++ b/docs/user/_sources/subject_list_config.txt @@ -35,7 +35,7 @@ or:: docker run -i --rm --user $(id -u) -v /path/to/data_config:/scratch -w="/scratch" fcpindi/c-pac:latest bids_dir outputs_dir cli -- utils data_config new_settings_template -If running a container directly (either of the latter options rather than the first option, which is through the :ref:`C-PAC Python package `), ``bids_dir`` and ``outputs_dir`` are required but not used, so any non-null string will work for those positional arguments. +If running a container directly (either of the latter options rather than the first option, which is through :doc:`cpac (Python package) `), ``bids_dir`` and ``outputs_dir`` are required but not used, so any non-null string will work for those positional arguments. The Singularity command will create ``data_settings.yml`` in the current working directory you call the command from. diff --git a/docs/user/_sources/utils/help.txt b/docs/user/_sources/utils/help.txt deleted file mode 100644 index 976b3c886..000000000 --- a/docs/user/_sources/utils/help.txt +++ /dev/null @@ -1,6 +0,0 @@ -Usage: cpac utils -````````````````` - -.. command-output:: cpac utils --help - :shell: - :ellipsis: 0,9 \ No newline at end of file