deploy: fcc3e97

.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 698ed1b428a3993142c7cbe7361855b8
+tags: d77d1c0d9ca2f4c8421862c7c5a0d620

_sources/advanced_recipe.rst.txt

@@ -0,0 +1,62 @@
+Advanced Container Recipes
+==========================
+
+.. objectives::
+
+   * Understand the complexities of building advanced scientific software stacks within containers.
+   * Learn to construct detailed Apptainer definition files that encapsulate comprehensive scientific environments.
+   * Develop proficiency in crafting container recipes that ensure reproducibility and consistency across different HPC systems.
+
+   This demo will walk you through the process of creating advanced container recipes using Apptainer, aimed at constructing robust scientific software environments. These environments often require multiple, intricately linked software tools and libraries that must be correctly configured to work together seamlessly.
+
+.. prerequisites::
+
+   * Access to an HPC system with Apptainer installed.
+   * Basic to intermediate knowledge of Linux, shell scripting, and scientific software installation.
+   * Understanding of dependency management and environment configuration for scientific computing.
+
+Building a scientific software stack in a container can be challenging due to dependencies and the need for specific configurations. By using advanced container recipes, researchers can encapsulate these environments, making them portable and reproducible. This is particularly important in scientific research, where precise control over software versions and settings is crucial for validating experimental results.
+
+First, we'll outline a container recipe that includes a typical setup for a bioinformatics analysis environment, featuring tools like BLAST, Python, and R.
+
+.. code-block:: bash
+
+   # Example Apptainer definition file for a bioinformatics stack
+   Bootstrap: library
+   From: ubuntu:20.04
+   
+   %post
+       apt-get update && apt-get install -y wget build-essential python3 python3-pip r-base
+       pip3 install numpy scipy pandas biopython
+       wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.10.1+-x64-linux.tar.gz
+       tar -xzf ncbi-blast-2.10.1+-x64-linux.tar.gz -C /usr/local/bin --strip-components=1
+       echo 'export PATH=/usr/local/bin:$PATH' >> ~/.bashrc
+
+   %environment
+       export PATH=/usr/local/bin:$PATH
+
+   %runscript
+       echo "Environment for bioinformatics analysis ready. Tools available: BLAST, Python, R."
+       exec /bin/bash
+   
+
+.. code-block:: bash
+   
+   # Build the container for the bioinformatics stack
+   apptainer build bioinfo_container.sif bioinfo.def
+
+
+This block constructs the ``bioinfo_container.sif`` from the ``bioinfo.def`` file. It installs critical tools for bioinformatics, including BLAST for sequence analysis, and a suite of Python and R libraries commonly used in data analysis and visualization.
+
+.. code-block:: bash
+   
+   # Run the container, providing an interactive shell
+   apptainer shell bioinfo_container.sif
+
+
+This command provides an interactive shell within the ``bioinfo_container.sif``, allowing users to execute the installed tools and perform analyses as if they were running on a native environment. This setup is ideal for ensuring that all users, regardless of their host system configuration, can reproduce the scientific computations.
+
+Summary
+-------
+In this tutorial, you've learned how to create advanced container recipes using Apptainer to build a comprehensive scientific software stack. This skill is invaluable for researchers who need to ensure the reproducibility of their experiments and for IT administrators tasked with maintaining consistent
+

_sources/advanced_running_containers.rst.txt

@@ -0,0 +1,135 @@
+Advanced settings for running containers
+========================================
+
+.. objectives::
+
+   * Learn more advanced flags for running containers
+
+Binding folders into your container
+-----------------------------------
+
+When you launch a program in a container, the program runs in a contained
+environment. The file system in the container might not have the same
+folders structure as the machine that runs the container.
+
+So, when you want to work on your data, you need to bring it with you into
+this contained world inside the container.
+
+This is done via method called **mount binding**.
+
+Binding means that a folder from the host system is mapped into a folder
+inside the container.
+
+.. figure:: img/default_mounts.png
+
+   Figure 1: Only some folders are mounted by default
+
+By default, ``$HOME``, ``$CWD`` (current working directory)
+``/tmp`` and
+`few other paths <https://apptainer.org/docs/user/main/bind_paths_and_mounts.html#system-defined-bind-paths>`__
+are bound to the image.
+
+If you want to do additional mappings you need to do it by giving an
+extra arguments to the command you're running.
+
+The following would bind folder ``/scratch`` from the host system to
+``/scratch`` in the container:
+
+.. code-block:: console
+
+   $ apptainer exec --bind /scratch example.sif ls /scratch
+
+.. figure:: img/bind_example.png
+
+   Figure 2: Binding a directory inside a container
+
+Setting ``--bind``-argument works for ``apptainer run``- and
+``apptainer shell``-commands as well.
+
+You can also bind directories to different places. This is especially
+helpful if, for example. the code in container expects that data
+should be in ``/data``:
+
+.. code-block:: console
+
+   $ apptainer exec --bind /scratch:/data example.sif ls /data
+
+.. warning::
+
+   Bind mounts **are the same folders inside and outside** of the image.
+
+   Deleting file from the folder that you bound inside the image will
+   delete the file completely.
+
+
+Running containers that use GPUs
+--------------------------------
+
+If your program uses GPUs, you'll need to make the GPUs visible in
+the container. This is done by giving additional flag to the
+apptainer command.
+
+Using NVIDIA's GPUs
+*******************
+
+When using NVIDIA's GPUs that use the CUDA-framework the flag is ``--nv``.
+
+As an example, let's get a CUDA-enabled PyTorch-image:
+
+.. code-block:: console
+
+   $ apptainer pull pytorch-cuda.sif docker://docker.io/pytorch/pytorch:2.2.2-cuda12.1-cudnn8-runtime
+
+Now when we launch the image, we can give the image GPU access with
+
+.. code-block:: console
+
+   $ apptainer exec --nv pytorch-cuda.sif python -c 'import torch; print(torch.cuda.is_available())'
+
+.. figure:: img/nv_example.png
+
+   Figure 3: Enabling NVIDIA's GPUs in containers
+
+.. admonition:: Expected result
+   :class: dropdown
+
+   If you run this in a system with an NVIDIA GPU, you should see the following result:
+
+   .. code-block:: console
+
+      $ apptainer exec --nv pytorch-cuda.sif python -c 'import torch; print(torch.cuda.is_available())'
+      True
+
+
+
+Using AMD's GPUs
+****************
+
+When using AMD's GPUs that use the ROCm-framework the flag is ``--rocm``.
+
+As an example, let's get a ROCm-enabled PyTorch-image:
+
+.. code-block:: console
+
+   $ apptainer pull pytorch-rocm.sif docker://docker.io/rocm/pytorch:rocm6.1_ubuntu22.04_py3.10_pytorch_2.1.2
+
+.. figure:: img/rocm_example.png
+
+   Figure 4: Enabling AMD's GPUs in containers
+
+Now when we launch the image, we can give the image GPU access with
+
+.. code-block:: console
+
+   $ apptainer exec --rocm pytorch-rocm.sif python -c 'import torch; print(torch.cuda.is_available())'
+
+.. admonition:: Expected result
+   :class: dropdown
+
+   If you run this in a system with an AMD GPU, you should see the following result:
+
+   .. code-block:: console
+
+      $ apptainer exec --rocm pytorch-rocm.sif python -c 'import torch; print(torch.cuda.is_available())'
+      True
+

_sources/apptainer_gpus.rst.txt

@@ -0,0 +1,64 @@
+Using GPUs with Apptainer
+=========================
+
+.. objectives::
+
+   * Understand how to access and utilize GPU resources within an Apptainer container.
+   * Learn to configure an Apptainer container for running applications that require GPU computation.
+   * Develop skills to integrate GPU-based processing within containerized scientific workflows.
+
+   This demo will guide you through the process of configuring and running an Apptainer container that utilizes GPUs, which is essential for high-performance computing tasks that require significant computational power, such as deep learning and large-scale data processing. Utilizing GPUs within containers can dramatically increase the efficiency and speed of these computations.
+
+.. prerequisites::
+
+   * Access to an HPC system with NVIDIA GPUs installed.
+   * Apptainer installed on the HPC system.
+   * Basic understanding of containerization and GPU computing principles.
+
+GPUs are powerful tools for accelerating computational workloads, particularly those involving parallel processing tasks. However, accessing GPU resources within a containerized environment presents unique challenges, such as ensuring proper drivers are installed and accessible to the container. This demo will show you how to overcome these challenges to harness the full potential of GPU computing in an HPC setting.
+
+First, we will create an Apptainer definition file that sets up an environment capable of utilizing NVIDIA GPUs.
+
+.. code-block:: bash
+
+   # Example Apptainer definition file for using NVIDIA GPUs
+   Bootstrap: library
+   From: nvcr.io/nvidia/cuda:11.0-base
+   
+   %post
+       apt-get update && apt-get install -y cuda-samples-11-0
+   
+   %test
+       cd /usr/local/cuda-11.0/samples/1_Utilities/deviceQuery
+       make
+       ./deviceQuery
+   
+   %environment
+       export PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
+       export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
+
+   %runscript
+       echo "Running CUDA device query..."
+       ./deviceQuery
+   
+
+.. code-block:: bash
+   
+   # Build the GPU-enabled container
+   apptainer build cuda_container.sif cuda.def
+   
+
+This block constructs the ``cuda_container.sif`` container from the ``cuda.def`` definition file, which includes the CUDA base image from NVIDIA's container registry. This setup ensures that the container will have access to the necessary CUDA libraries and tools to utilize GPU resources.
+
+.. code-block:: bash
+   
+   # Run the container with GPU support
+   apptainer exec --nv cuda_container.sif /bin/bash
+
+
+This command executes the container with the ``--nv`` flag, which enables NVIDIA GPU support within the container. This flag is crucial as it allows the container to access the host's GPU resources, essential for running GPU-accelerated applications.
+
+
+Summary
+-------
+In this tutorial, you have learned how to configure and use an Apptainer container to access and utilize GPU resources for high-performance computational tasks. This capability is particularly valuable in scientific computing, where the processing power of GPUs can be leveraged to accelerate research and development workflows. By integrating GPU support into your containers, you enhance their functionality and applicability in a wide range of HPC scenarios.