deploy: ea35650

.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: df55cf7a91f7fdbecf0348b70949546e
+tags: d77d1c0d9ca2f4c8421862c7c5a0d620

_sources/advanced_recipe.rst.txt

@@ -0,0 +1,62 @@
+Advanced Container Recipes
+==========================
+
+.. objectives::
+
+   * Understand the complexities of building advanced scientific software stacks within containers.
+   * Learn to construct detailed Apptainer definition files that encapsulate comprehensive scientific environments.
+   * Develop proficiency in crafting container recipes that ensure reproducibility and consistency across different HPC systems.
+
+   This demo will walk you through the process of creating advanced container recipes using Apptainer, aimed at constructing robust scientific software environments. These environments often require multiple, intricately linked software tools and libraries that must be correctly configured to work together seamlessly.
+
+.. prerequisites::
+
+   * Access to an HPC system with Apptainer installed.
+   * Basic to intermediate knowledge of Linux, shell scripting, and scientific software installation.
+   * Understanding of dependency management and environment configuration for scientific computing.
+
+Building a scientific software stack in a container can be challenging due to dependencies and the need for specific configurations. By using advanced container recipes, researchers can encapsulate these environments, making them portable and reproducible. This is particularly important in scientific research, where precise control over software versions and settings is crucial for validating experimental results.
+
+First, we'll outline a container recipe that includes a typical setup for a bioinformatics analysis environment, featuring tools like BLAST, Python, and R.
+
+.. code-block:: bash
+
+   # Example Apptainer definition file for a bioinformatics stack
+   Bootstrap: library
+   From: ubuntu:20.04
+   
+   %post
+       apt-get update && apt-get install -y wget build-essential python3 python3-pip r-base
+       pip3 install numpy scipy pandas biopython
+       wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.10.1+-x64-linux.tar.gz
+       tar -xzf ncbi-blast-2.10.1+-x64-linux.tar.gz -C /usr/local/bin --strip-components=1
+       echo 'export PATH=/usr/local/bin:$PATH' >> ~/.bashrc
+
+   %environment
+       export PATH=/usr/local/bin:$PATH
+
+   %runscript
+       echo "Environment for bioinformatics analysis ready. Tools available: BLAST, Python, R."
+       exec /bin/bash
+   
+
+.. code-block:: bash
+   
+   # Build the container for the bioinformatics stack
+   apptainer build bioinfo_container.sif bioinfo.def
+
+
+This block constructs the ``bioinfo_container.sif`` from the ``bioinfo.def`` file. It installs critical tools for bioinformatics, including BLAST for sequence analysis, and a suite of Python and R libraries commonly used in data analysis and visualization.
+
+.. code-block:: bash
+   
+   # Run the container, providing an interactive shell
+   apptainer shell bioinfo_container.sif
+
+
+This command provides an interactive shell within the ``bioinfo_container.sif``, allowing users to execute the installed tools and perform analyses as if they were running on a native environment. This setup is ideal for ensuring that all users, regardless of their host system configuration, can reproduce the scientific computations.
+
+Summary
+-------
+In this tutorial, you've learned how to create advanced container recipes using Apptainer to build a comprehensive scientific software stack. This skill is invaluable for researchers who need to ensure the reproducibility of their experiments and for IT administrators tasked with maintaining consistent
+

_sources/apptainer_gpus.rst.txt

@@ -0,0 +1,64 @@
+Using GPUs with Apptainer
+=========================
+
+.. objectives::
+
+   * Understand how to access and utilize GPU resources within an Apptainer container.
+   * Learn to configure an Apptainer container for running applications that require GPU computation.
+   * Develop skills to integrate GPU-based processing within containerized scientific workflows.
+
+   This demo will guide you through the process of configuring and running an Apptainer container that utilizes GPUs, which is essential for high-performance computing tasks that require significant computational power, such as deep learning and large-scale data processing. Utilizing GPUs within containers can dramatically increase the efficiency and speed of these computations.
+
+.. prerequisites::
+
+   * Access to an HPC system with NVIDIA GPUs installed.
+   * Apptainer installed on the HPC system.
+   * Basic understanding of containerization and GPU computing principles.
+
+GPUs are powerful tools for accelerating computational workloads, particularly those involving parallel processing tasks. However, accessing GPU resources within a containerized environment presents unique challenges, such as ensuring proper drivers are installed and accessible to the container. This demo will show you how to overcome these challenges to harness the full potential of GPU computing in an HPC setting.
+
+First, we will create an Apptainer definition file that sets up an environment capable of utilizing NVIDIA GPUs.
+
+.. code-block:: bash
+
+   # Example Apptainer definition file for using NVIDIA GPUs
+   Bootstrap: library
+   From: nvcr.io/nvidia/cuda:11.0-base
+   
+   %post
+       apt-get update && apt-get install -y cuda-samples-11-0
+   
+   %test
+       cd /usr/local/cuda-11.0/samples/1_Utilities/deviceQuery
+       make
+       ./deviceQuery
+   
+   %environment
+       export PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
+       export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
+
+   %runscript
+       echo "Running CUDA device query..."
+       ./deviceQuery
+   
+
+.. code-block:: bash
+   
+   # Build the GPU-enabled container
+   apptainer build cuda_container.sif cuda.def
+   
+
+This block constructs the ``cuda_container.sif`` container from the ``cuda.def`` definition file, which includes the CUDA base image from NVIDIA's container registry. This setup ensures that the container will have access to the necessary CUDA libraries and tools to utilize GPU resources.
+
+.. code-block:: bash
+   
+   # Run the container with GPU support
+   apptainer exec --nv cuda_container.sif /bin/bash
+
+
+This command executes the container with the ``--nv`` flag, which enables NVIDIA GPU support within the container. This flag is crucial as it allows the container to access the host's GPU resources, essential for running GPU-accelerated applications.
+
+
+Summary
+-------
+In this tutorial, you have learned how to configure and use an Apptainer container to access and utilize GPU resources for high-performance computational tasks. This capability is particularly valuable in scientific computing, where the processing power of GPUs can be leveraged to accelerate research and development workflows. By integrating GPU support into your containers, you enhance their functionality and applicability in a wide range of HPC scenarios.

_sources/basics_running_containers.rst.txt

@@ -0,0 +1,183 @@
+Basics of running containers
+============================
+
+.. objectives::
+
+   * Learn what methods are available to run a container
+
+
+Different ways you can run your containers
+------------------------------------------
+
+You can run your container in multiple different ways:
+
+- You can run the container as a single application
+- You can get an interactive terminal in the container
+- You can run an application from within the container
+
+
+We'll cover these ways in this individual sections.
+
+
+How to read Apptainer commands?
+-------------------------------
+
+Apptainer's commands can sometimes have multiple flags
+which can make it hard to parse the command line.
+
+A typical apptainer command has:
+
+1. ``apptainer`` is at the start because we use the
+   Apptainer client to run the commands.
+2. A subcommand (like ``pull``, ``build``, ``exec``, ``run`` etc.)
+   that tells which Apptainer feature we want to use.
+3. Often after the subcommand we have the name of the image we
+   want to use.
+4. At the end we might have additional commands that Apptainer
+   ignores.
+
+For legibility reasons many commands in this lesson are highlighted
+in this format:
+
+.. figure:: img/apptainer_example.png
+   :width: 100%
+
+   Figure 1: How apptainer-commands are highlighted
+
+
+For the comprehensive list of client commands, see
+`Apptainer CLI reference page <https://apptainer.org/docs/user/latest/cli.html>`__.
+
+
+Obtaining a container from container registry
+---------------------------------------------
+
+For these examples let's use a
+`Python image from Docker Hub <https://hub.docker.com/_/python>`__.
+
+For now let's just consider it as an application we want
+to use and that this application is available in Docker Hub.
+
+Let's create the container image with the following command:
+
+.. code-block:: console
+
+   $ apptainer pull python.sif docker://python
+
+We'll talk about container images and image building later on.
+For now it is enough to know that the file ``python.sif`` contains
+a Python installation.
+
+Running the container
+---------------------
+
+Running the container means that we run some specific program
+that the image creator has decided to be the main program for
+this image.
+
+When we run the container we will execute a so-called entry script
+or runscript that contains commands specified by the image creators.
+
+.. warning::
+
+   Remember to only run containers from sources that you trust.
+
+Let's run the Python container image:
+
+.. code-block:: console
+
+   $ apptainer run python.sif
+
+.. admonition:: Expected result
+   :class: dropdown
+
+   We get a Python interpreter running in the container:
+
+   .. code-block:: console
+
+      $ apptainer run python.sif 
+      Python 3.12.3 (main, Apr 10 2024, 14:35:18) [GCC 12.2.0] on linux
+      Type "help", "copyright", "credits" or "license" for more information.
+      >>> 
+
+
+You should read the command line like this:
+
+.. figure:: img/run_example.png
+   :scale: 50%
+
+   Figure 2: A simple run command
+
+This syntax might seem self-evident, but in more complex use cases we will give
+additional flags to various parts of the command.
+
+
+Launching an interactive shell in the container
+-----------------------------------------------
+
+We can launch an interactive shell in a container with
+``apptainer shell``-command:
+
+.. code-block:: console
+
+   $ apptainer shell python.sif
+
+.. admonition:: Expected result
+   :class: dropdown
+
+   We get a ``bash`` terminal running in the container:
+
+   .. code-block:: console
+
+      $ apptainer shell python.sif 
+      Apptainer> echo $SHELL
+      /bin/bash
+
+You should read the command line like this:
+
+.. figure:: img/shell_example.png
+   :scale: 50%
+
+   Figure 3: Launch a shell in a container
+
+
+Running a single program from the container
+-------------------------------------------
+
+We can launch a single program in a container with
+``apptainer exec``-command.
+
+The Python container is based on a Debian Linux distribution.
+Let's check the Debian version in the container:
+
+.. code-block:: console
+
+   $ apptainer exec python.sif cat /etc/debian_version
+
+.. admonition:: Expected result
+   :class: dropdown
+
+   With the command we print the contents of the
+   ``/etc/debian_version``-file:
+
+   .. code-block:: console
+
+      $ apptainer exec python.sif cat /etc/debian_version
+      12.5
+
+You should read the command line like this:
+
+.. figure:: img/exec_example.png
+   :scale: 50%
+
+   Figure 4: Launch a program in a container
+
+
+Review of this session
+----------------------
+
+.. admonition:: Key points to remember
+
+   - You can run the container with ``apptainer run my_image.sif``
+   - You can start an interactive shell in a container with ``apptainer shell my_image.sif``
+   - You can run a single program in a container with ``apptainer exec my_image.sif my_program arg1 arg2``

_sources/binding_folders.rst.txt

@@ -0,0 +1,68 @@
+Binding folders into your container
+===================================
+
+.. objectives::
+
+   * Learn how you can bind folders into containers
+
+When you launch a program in a container, the program runs in a contained
+environment. The file system in the container might not have the same
+folders structure as the machine that runs the container.
+
+So, when you want to work on your data, you need to bring it with you into
+this contained world inside the container.
+
+This is done via method called **mount binding**.
+
+Binding means that a folder from the host system is mapped into a folder
+inside the container.
+
+.. figure:: img/default_mounts.png
+
+   Figure 1: Only some folders are mounted by default
+
+By default, ``$HOME``, ``$CWD`` (current working directory)
+``/tmp`` and
+`few other paths <https://apptainer.org/docs/user/main/bind_paths_and_mounts.html#system-defined-bind-paths>`__
+are bound to the image.
+
+If you want to do additional mappings you need to do it by giving an
+extra arguments to the command you're running.
+
+The following would bind folder ``/scratch`` from the host system to
+``/scratch`` in the container:
+
+.. code-block:: console
+
+   $ apptainer exec --bind /scratch example.sif ls /scratch
+
+.. figure:: img/bind_example.png
+
+   Figure 2: Binding a directory inside a container
+
+Setting ``--bind``-argument works for ``apptainer run``- and
+``apptainer shell``-commands as well.
+
+You can also bind directories to different places. This is especially
+helpful if, for example. the code in container expects that data
+should be in ``/data``:
+
+.. code-block:: console
+
+   $ apptainer exec --bind /scratch:/data example.sif ls /data
+
+.. warning::
+
+   Bind mounts **are the same folders inside and outside** of the image.
+
+   Deleting file from the folder that you bound inside the image will
+   delete the file completely.
+
+.. admonition:: Key points to remember
+
+   - By default only some folders are bound inside the container.
+   - You'll need to manually ``--bind``-folders from outside of
+     the container if you want to use them inside the container.
+   - Bound folders are the same folders inside and outside of the
+     container.
+