From d5f539c66fa5f59f4ef19c6675e49e9b9078c8f0 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Tue, 21 Feb 2023 16:57:37 -0500 Subject: [PATCH 1/4] :memo: Document multistage builds --- docs/_sources/developer/installation.rst | 95 +++++++++++++++++++-- docs/_sources/references/Exported Items.bib | 2 - docs/_sources/references/references.bib | 19 +++++ 3 files changed, 107 insertions(+), 9 deletions(-) delete mode 100644 docs/_sources/references/Exported Items.bib diff --git a/docs/_sources/developer/installation.rst b/docs/_sources/developer/installation.rst index 10b6d376f..706f60da9 100644 --- a/docs/_sources/developer/installation.rst +++ b/docs/_sources/developer/installation.rst @@ -1,14 +1,95 @@ .. _installation: +**************** +Installing C-PAC +**************** +.. _install_cpac: -*************** -Installing CPAC -*************** +Downloading C-PAC +================= -.. _install_cpac: +C-PAC is only supported as a container image. To get the latest version of C-PAC, use one of the following commands, depending on your containerization software: + +Docker +^^^^^^ + +.. code:: BASH + + docker pull fcpindi/c-pac:latest + +Apptainer / Singularity +^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: BASH + + singularity pull C-PAC_latest.sif docker://fcpindi/c-pac:latest + +other versions +^^^^^^^^^^^^^^ + +For a development image of the next release (the latest build of |develop|_), simply replace ``latest`` with ``nightly``. + +For branch-specific images, replace ``fcpindi/c-pac`` with ``ghcr.io/fcp-indi/c-pac`` and replace ``latest`` with the appropriate branch-specific tag. You can see the GitHub Actions-generated image tags at https://github.com/FCP-INDI/C-PAC/pkgs/container/c-pac or in the log of the relevent GitHub Actions "Build and test C-PAC job" under the heading "Set tag & Dockerfile". + +.. |develop| replace:: ``develop`` + +.. _develop: https://github.com/FCP-INDI/C-PAC/tree/develop + +Replacing C-PAC code in an already-built image +============================================== + +If you have made changes to C-PAC that you would like to test without rebuilding the whole image, you can simply bind your code to your container when you launch to override the internal copy of C-PAC like + +.. code:: BASH + + docker run --rm -it \ + -v ${LOCAL_CPAC_PATH}/CPAC:/code/CPAC \ + -v ${LOCAL_CPAC_PATH}/dev/docker_data/run.py:/code/run.py \ + -v ${LOCAL_CPAC_PATH}/dev/docker_data/run-with-freesurfer.sh:/code/run-with-freesurfer.sh \ + fcpindi/c-pac:latest \ + ${INPUT_BIDS_DIR} ${OUTPUT_DIR} participant + +or + +.. code:: BASH + + singularity run --cleanenv \ + -B ${LOCAL_CPAC_PATH}/CPAC:/code/CPAC \ + -B ${LOCAL_CPAC_PATH}/dev/docker_data/run.py:/code/run.py \ + -B ${LOCAL_CPAC_PATH}/dev/docker_data/run-with-freesurfer.sh:/code/run-with-freesurfer.sh \ + C-PAC_latest.sif \ + ${INPUT_BIDS_DIR} ${OUTPUT_DIR} participant + +where ``${LOCAL_CPAC_PATH}`` is the path to the root of your clone of the C-PAC git repository. If you haven't made any changes to the entrypoint scripts (``run.py`` / ``run-with-freesurfer.sh``), then ``${LOCAL_CPAC_PATH}/CPAC:/code/CPAC`` is the only binding you need to replace the in-container C-PAC code with your local copy. + +Building a new C-PAC image +========================== + +The C-PAC build process makes use of Docker's multi-stage build functionality :footcite:`Mult23` to reduce iteration time. See https://github.com/FCP-INDI/C-PAC/tree/develop/.github for the latest Dockerfiles and GitHub Actions workflow configurations. + +For each stage that you don't need to change, you can just pull that stage from GitHub Packages :footcite:`Work23`. If you need to build a stage, you can do so locally by running + +.. code:: BASH + + cd ${LOCAL_CPAC_PATH} + docker build \ + --platform linux/amd64 \ + -f .github/Dockerfiles/${STAGE_DOCKERFILE} \ + -t ghcr.io/fcpindi/c-pac/${STAGE}:${TAG} \ + . + +To interactively develop a staging image Dockerfile, attach to a base image in a BASH session. For some staging images, BASH is not installed (e.g., images built ``FROM scratch``, so you may want to interact with a different image than what will ultimately be packaged. The base image in a Dockerfile is defined in the final ``FROM`` instruction. For example + +.. code:: BASH + + docker run --rm -it \ + --entrypoint /bin/bash \ + ghcr.io/fcp-indi/c-pac/ubuntu:jammy-non-free + +enters a C-PAC staging Ubuntu 22.04 image where you can test installation commands interactively before committing them in a Dockerfile. -Downloading the latest version of CPAC -====================================== +References +========== -This section describes how to download the latest developer version of CPAC +.. footbibliography:: diff --git a/docs/_sources/references/Exported Items.bib b/docs/_sources/references/Exported Items.bib deleted file mode 100644 index 139597f9c..000000000 --- a/docs/_sources/references/Exported Items.bib +++ /dev/null @@ -1,2 +0,0 @@ - - diff --git a/docs/_sources/references/references.bib b/docs/_sources/references/references.bib index cc23815f3..aac81b08d 100644 --- a/docs/_sources/references/references.bib +++ b/docs/_sources/references/references.bib @@ -145,6 +145,16 @@ @misc{Meji15 year = {2015} } +@misc{Mult23, + title = {Multi-Stage Builds}, + year = {2023}, + month = feb, + journal = {Docker Documentation}, + abstract = {Keeping your images small with multi-stage builds}, + url = {https://docs.docker.com/build/building/multi-stage/}, + langid = {english} +} + @misc{Neur, title = {{{NeuroData}}'s {{MRI}} to {{Graphs}}}, author = {{NeuroData}}, @@ -250,6 +260,15 @@ @article{Wang21a pmid = {33789137} } +@misc{Work23, + title = {Working with the {{Container}} Registry}, + year = {2023}, + journal = {GitHub Docs}, + abstract = {You can store and manage Docker and OCI images in the Container registry, which uses the package namespace https://ghcr.io.}, + url = {https://ghdocs-prod.azurewebsites.net/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry}, + langid = {english} +} + @article{Xu19, title = {Interindividual Variability of Functional Connectivity in Awake and Anesthetized Rhesus Macaque Monkeys}, author = {Xu, Ting and Sturgeon, Darrick and Ramirez, Julian S. B. and {Froudist-Walsh}, Se{\'a}n and Margulies, Daniel S. and Schroeder, Charles E. and Fair, Damien A. and Milham, Michael P.}, From d84ac7d2b3535e04c07b33e8de2f154ef0a2ddd6 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Wed, 22 Feb 2023 11:39:15 -0500 Subject: [PATCH 2/4] :bug: Fix GH Docs link --- docs/_sources/references/references.bib | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/_sources/references/references.bib b/docs/_sources/references/references.bib index aac81b08d..1b25494c8 100644 --- a/docs/_sources/references/references.bib +++ b/docs/_sources/references/references.bib @@ -265,7 +265,7 @@ @misc{Work23 year = {2023}, journal = {GitHub Docs}, abstract = {You can store and manage Docker and OCI images in the Container registry, which uses the package namespace https://ghcr.io.}, - url = {https://ghdocs-prod.azurewebsites.net/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry}, + url = {https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry}, langid = {english} } From 1243eb80f0a835269c54f2f0e92c874e0b871938 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Wed, 22 Feb 2023 11:49:37 -0500 Subject: [PATCH 3/4] :memo: Add link to C-PAC staging images in GHCR --- docs/_sources/developer/installation.rst | 2 +- docs/_sources/references/references.bib | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/_sources/developer/installation.rst b/docs/_sources/developer/installation.rst index 706f60da9..556cd4358 100644 --- a/docs/_sources/developer/installation.rst +++ b/docs/_sources/developer/installation.rst @@ -68,7 +68,7 @@ Building a new C-PAC image The C-PAC build process makes use of Docker's multi-stage build functionality :footcite:`Mult23` to reduce iteration time. See https://github.com/FCP-INDI/C-PAC/tree/develop/.github for the latest Dockerfiles and GitHub Actions workflow configurations. -For each stage that you don't need to change, you can just pull that stage from GitHub Packages :footcite:`Work23`. If you need to build a stage, you can do so locally by running +For each stage that you don't need to change, you can just pull that stage from GitHub Packages :footcite:`Work23,GHCR`. If you need to build a stage, you can do so locally by running .. code:: BASH diff --git a/docs/_sources/references/references.bib b/docs/_sources/references/references.bib index 1b25494c8..c600578b5 100644 --- a/docs/_sources/references/references.bib +++ b/docs/_sources/references/references.bib @@ -86,6 +86,13 @@ @book{fMRI16 url = {https://fmriprep.org/en/stable/} } +@misc{GHCR, + title = {C-{{PAC}}}, + author = {{FCP-INDI}}, + journal = {GitHub Packages}, + url = {https://github.com/orgs/FCP-INDI/packages?repo\_name=C-PAC} +} + @misc{Habe18, title = {Convert from {CIFTI2} ({HCP}) to {3D} array}, url = {https://neurostars.org/t/convert-from-cifti2-hcp-to-3d-array/3054}, From 4a6df41f9b43186e778a85e95c162889676adde1 Mon Sep 17 00:00:00 2001 From: Jon Clucas Date: Wed, 22 Feb 2023 11:52:04 -0500 Subject: [PATCH 4/4] :memo: Set user when interacting --- docs/_sources/developer/installation.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/_sources/developer/installation.rst b/docs/_sources/developer/installation.rst index 556cd4358..2a464d57e 100644 --- a/docs/_sources/developer/installation.rst +++ b/docs/_sources/developer/installation.rst @@ -85,6 +85,7 @@ To interactively develop a staging image Dockerfile, attach to a base image in a docker run --rm -it \ --entrypoint /bin/bash \ + --user root \ ghcr.io/fcp-indi/c-pac/ubuntu:jammy-non-free enters a C-PAC staging Ubuntu 22.04 image where you can test installation commands interactively before committing them in a Dockerfile.