initial commit

content/advanced_recipe.rst

@@ -0,0 +1,60 @@
+Advanced Container Recipes
+==========================
+
+.. objectives::
+
+   * Understand the complexities of building advanced scientific software stacks within containers.
+   * Learn to construct detailed Apptainer definition files that encapsulate comprehensive scientific environments.
+   * Develop proficiency in crafting container recipes that ensure reproducibility and consistency across different HPC systems.
+
+   This demo will walk you through the process of creating advanced container recipes using Apptainer, aimed at constructing robust scientific software environments. These environments often require multiple, intricately linked software tools and libraries that must be correctly configured to work together seamlessly.
+
+.. prerequisites::
+
+   * Access to an HPC system with Apptainer installed.
+   * Basic to intermediate knowledge of Linux, shell scripting, and scientific software installation.
+   * Understanding of dependency management and environment configuration for scientific computing.
+
+Building a scientific software stack in a container can be challenging due to dependencies and the need for specific configurations. By using advanced container recipes, researchers can encapsulate these environments, making them portable and reproducible. This is particularly important in scientific research, where precise control over software versions and settings is crucial for validating experimental results.
+
+First, we'll outline a container recipe that includes a typical setup for a bioinformatics analysis environment, featuring tools like BLAST, Python, and R.
+
+``` bash
+# Example Apptainer definition file for a bioinformatics stack
+Bootstrap: library
+From: ubuntu:20.04
+
+%post
+    apt-get update && apt-get install -y wget build-essential python3 python3-pip r-base
+    pip3 install numpy scipy pandas biopython
+    wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.10.1+-x64-linux.tar.gz
+    tar -xzf ncbi-blast-2.10.1+-x64-linux.tar.gz -C /usr/local/bin --strip-components=1
+    echo 'export PATH=/usr/local/bin:$PATH' >> ~/.bashrc
+
+%environment
+    export PATH=/usr/local/bin:$PATH
+
+%runscript
+    echo "Environment for bioinformatics analysis ready. Tools available: BLAST, Python, R."
+    exec /bin/bash
+
+```
+
+``` bash
+# Build the container for the bioinformatics stack
+apptainer build bioinfo_container.sif bioinfo.def
+```
+
+This block constructs the `bioinfo_container.sif` from the `bioinfo.def` file. It installs critical tools for bioinformatics, including BLAST for sequence analysis, and a suite of Python and R libraries commonly used in data analysis and visualization.
+
+``` bash
+# Run the container, providing an interactive shell
+apptainer shell bioinfo_container.sif
+```
+
+This command provides an interactive shell within the `bioinfo_container.sif`, allowing users to execute the installed tools and perform analyses as if they were running on a native environment. This setup is ideal for ensuring that all users, regardless of their host system configuration, can reproduce the scientific computations.
+
+Summary
+-------
+In this tutorial, you've learned how to create advanced container recipes using Apptainer to build a comprehensive scientific software stack. This skill is invaluable for researchers who need to ensure the reproducibility of their experiments and for IT administrators tasked with maintaining consistent
+

content/apptainer_gpus.rst

@@ -0,0 +1,62 @@
+Using GPUs with Apptainer
+=========================
+
+.. objectives::
+
+   * Understand how to access and utilize GPU resources within an Apptainer container.
+   * Learn to configure an Apptainer container for running applications that require GPU computation.
+   * Develop skills to integrate GPU-based processing within containerized scientific workflows.
+
+   This demo will guide you through the process of configuring and running an Apptainer container that utilizes GPUs, which is essential for high-performance computing tasks that require significant computational power, such as deep learning and large-scale data processing. Utilizing GPUs within containers can dramatically increase the efficiency and speed of these computations.
+
+.. prerequisites::
+
+   * Access to an HPC system with NVIDIA GPUs installed.
+   * Apptainer installed on the HPC system.
+   * Basic understanding of containerization and GPU computing principles.
+
+GPUs are powerful tools for accelerating computational workloads, particularly those involving parallel processing tasks. However, accessing GPU resources within a containerized environment presents unique challenges, such as ensuring proper drivers are installed and accessible to the container. This demo will show you how to overcome these challenges to harness the full potential of GPU computing in an HPC setting.
+
+First, we will create an Apptainer definition file that sets up an environment capable of utilizing NVIDIA GPUs.
+
+``` bash
+# Example Apptainer definition file for using NVIDIA GPUs
+Bootstrap: library
+From: nvcr.io/nvidia/cuda:11.0-base
+
+%post
+    apt-get update && apt-get install -y cuda-samples-11-0
+
+%test
+    cd /usr/local/cuda-11.0/samples/1_Utilities/deviceQuery
+    make
+    ./deviceQuery
+
+%environment
+    export PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
+    export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
+
+%runscript
+    echo "Running CUDA device query..."
+    ./deviceQuery
+
+```
+
+``` bash
+# Build the GPU-enabled container
+apptainer build cuda_container.sif cuda.def
+```
+
+This block constructs the `cuda_container.sif` container from the `cuda.def` definition file, which includes the CUDA base image from NVIDIA's container registry. This setup ensures that the container will have access to the necessary CUDA libraries and tools to utilize GPU resources.
+
+``` bash
+# Run the container with GPU support
+apptainer exec --nv cuda_container.sif /bin/bash
+```
+
+This command executes the container with the `--nv` flag, which enables NVIDIA GPU support within the container. This flag is crucial as it allows the container to access the host's GPU resources, essential for running GPU-accelerated applications.
+
+Summary
+-------
+In this tutorial, you have learned how to configure and use an Apptainer container to access and utilize GPU resources for high-performance computational tasks. This capability is particularly valuable in scientific computing, where the processing power of GPUs can be leveraged to accelerate research and development workflows. By integrating GPU support into your containers, you enhance their functionality and applicability in a wide range of HPC scenarios.
+

content/conda.rst

@@ -0,0 +1,57 @@
+Reproducibility with Python
+===========================
+
+.. objectives::
+
+   * Understand the role of Conda environments in ensuring reproducibility in scientific computing.
+   * Learn how to set up a Conda environment within an Apptainer container.
+   * Develop skills to create and manage Python environments that are consistent across various computing platforms.
+
+   This demo demonstrates the creation of a Conda environment within an Apptainer container to enhance reproducibility in scientific computing. Conda is a popular package management system that simplifies package installation and environment management. This is crucial in scientific research where experiments often require specific versions of software and libraries to ensure consistent results.
+
+.. prerequisites::
+
+   * Access to an HPC system with Apptainer installed.
+   * Basic knowledge of Python, Conda, and container technologies.
+
+Reproducibility is a cornerstone of scientific research, ensuring that results are consistent over time and across different platforms. By using Conda within Apptainer containers, researchers can encapsulate their computational environment, making their workflows portable and reproducible.
+
+First, we'll define a container with a Conda environment set up for a typical scientific Python stack.
+
+``` bash
+# Example Apptainer definition file for setting up a Conda environment
+Bootstrap: library
+From: continuumio/miniconda3
+
+%post
+    conda create -n scienv python=3.8 numpy scipy matplotlib ipython -y
+    echo "source activate scienv" >> ~/.bashrc
+
+%environment
+    export PATH=/opt/conda/envs/scienv/bin:$PATH
+
+%runscript
+    echo "Activating Conda environment and launching IPython..."
+    source activate scienv
+    ipython
+
+```
+
+``` bash
+# Build the container that includes the Conda environment
+apptainer build conda_container.sif conda.def
+```
+
+This block builds the `conda_container.sif` from the `conda.def` definition file, which uses the Miniconda image to install a Conda environment named `scienv` with several essential scientific computing packages like NumPy, SciPy, and Matplotlib.
+
+``` bash
+# Run the container, initiating the Conda environment
+apptainer exec conda_container.sif /bin/bash -c "source ~/.bashrc && ipython"
+```
+
+This command starts a shell that activates the Conda environment and launches IPython, a powerful interactive shell for Python. This setup allows users to interactively compute and visualize data within the consistent environment provided by the container.
+
+Summary
+-------
+In this tutorial, you've learned how to create a reproducible Python computing environment using Conda within an Apptainer container. This approach is invaluable for scientific researchers who need to ensure that their computational experiments are consistent and reproducible, regardless of the underlying hardware or software environment. By leveraging Conda and Apptainer, you can significantly enhance the portability and reliability of your scientific workflows.
+

content/debugging.rst

@@ -0,0 +1,64 @@
+Debugging Containers
+====================
+
+.. objectives::
+
+   * Understand the importance of being able to debug applications within container environments.
+   * Learn to enter an Apptainer container using a shell for debugging purposes.
+   * Develop practical troubleshooting skills for containerized applications in an HPC setting.
+
+   This demo will instruct on how to access an Apptainer container with a shell interface to perform debugging and troubleshooting tasks. Debugging within containers is crucial because it allows developers and administrators to directly interact with the running environment and resolve issues that may not be apparent from outside the container.
+
+.. prerequisites::
+
+   * Access to an HPC system with Apptainer installed.
+   * A container with a misbehaving application or one prepared for debugging.
+   * Basic familiarity with command-line interface operations and debugging tools.
+
+Containers encapsulate dependencies and environments but can sometimes obscure problems from the host system. By entering the container itself, you can use standard debugging tools and inspect the state of the application as it runs within its isolated environment.
+
+First, we'll set up a simple container that includes tools and an application setup conducive to debugging. Then, we'll demonstrate how to enter this container to troubleshoot issues.
+
+``` bash
+# Example Apptainer definition file for a debuggable environment
+Bootstrap: library
+From: ubuntu:20.04
+
+%post
+    apt-get update && apt-get install -y python3 python3-pip gdb vim
+    pip3 install flask
+    echo "from flask import Flask" > /app.py
+    echo "app = Flask(__name__)" >> /app.py
+    echo "@app.route('/')" >> /app.py
+    echo "def hello():" >> /app.py
+    echo "    raise Exception('Intentional Error for Debugging')" >> /app.py
+    echo "if __name__ == '__main__':" >> /app.py
+    echo "    app.run(host='0.0.0.0', port=5000, debug=True)" >> /app.py
+
+%environment
+    export FLASK_APP=/app.py
+
+%runscript
+    echo "Running Flask app with intentional errors for debugging..."
+    python3 /app.py
+
+```
+
+``` bash
+# Build the container for debugging
+apptainer build debug_container.sif debug.def
+```
+
+This block constructs the `debug_container.sif` from the `debug.def` file, which sets up a Flask application programmed to raise an exception. This setup is useful for demonstrating how to handle errors within a live application.
+
+``` bash
+# Enter the container with a shell to start debugging
+apptainer shell debug_container.sif
+```
+
+This command launches a shell within the `debug_container.sif`, allowing you to interact directly with the container's environment. Inside the container, you can run the application, use tools like `gdb` or `vim` to inspect and modify files, or check logs to diagnose issues.
+
+Summary
+-------
+In this tutorial, you've learned how to enter an Apptainer container with a shell for the purpose of debugging applications. This skill is essential for software development and system administration in containerized environments, particularly in HPC where applications may behave differently due to complex interactions with underlying hardware and software layers. By mastering container debugging, you can significantly improve the reliability and performance of your applications.
+

content/env_scripts.rst

@@ -0,0 +1,56 @@
+Environment Variables and Scripts
+=================================
+
+.. objectives::
+
+   * Understand the role of environment variables in configuring software behavior inside containers.
+   * Learn to set and export environment variables within an Apptainer container.
+   * Develop the ability to customize the behavior of applications running in containers through environmental configuration.
+
+   This demo will show you how to set and manipulate environment variables inside an Apptainer container. Environment variables are a critical component in software development and deployment, allowing you to modify the behavior of applications without changing code. They are especially useful in containerized environments where they can be used to control and customize settings across different deployments seamlessly.
+
+.. prerequisites::
+
+   * Access to an HPC system with Apptainer installed.
+   * Basic familiarity with shell scripting and command line operations.
+
+Setting environment variables inside a container is an essential skill for software developers and system administrators. These variables can dictate operational parameters such as database connections, logging levels, and other configurable settings that an application might need to operate correctly in different environments. In this demo, you will learn how to set these variables within an Apptainer container, ensuring your applications behave as expected under various conditions.
+
+First, we will create a simple Apptainer definition file that includes the necessary commands to set environment variables. Then, we will build and run the container to see how these variables affect the application's behavior.
+
+``` bash
+# Example Apptainer definition file setting environment variables
+Bootstrap: library
+From: ubuntu:20.04
+
+%post
+    apt-get update && apt-get install -y vim
+    echo 'export MY_VAR="Hello from Apptainer!"' >> /etc/profile
+
+%environment
+    export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH"
+    export ANOTHER_VAR="Another example variable"
+
+%runscript
+    echo "MY_VAR is set to $MY_VAR"
+    echo "ANOTHER_VAR is set to $ANOTHER_VAR"
+
+```
+
+``` bash
+# Build the container with environment variables
+apptainer build env_container.sif env.def
+```
+
+This block creates the `env_container.sif` container from the `env.def` file. It sets up an Ubuntu-based environment and installs Vim as an example application. The `%post` section modifies the system profile to permanently set an environment variable, while `%environment` sets it specifically for when the container is run.
+
+``` bash
+# Run the container to see environment variables in action
+apptainer run env_container.sif
+```
+
+This final command executes the container, running the script defined in `%runscript`. It outputs the values of the environment variables set during the build phase and the runtime environment, demonstrating how these can be manipulated within the container.
+
+Summary
+-------
+In this tutorial, you've learned how to set and export environment variables within an Apptainer container to influence the behavior of contained software applications. This skill is vital for managing software configurations in a reproducible and scalable manner across various computing environments. By using environment variables effectively, you can ensure that your applications perform consistently no matter where they are deployed.

content/first_build.rst

@@ -0,0 +1,50 @@
+Building Your First Container
+=============================
+
+.. objectives::
+
+   * Understand the basic concept and utility of containerization in HPC environments.
+   * Learn to create an Apptainer container using a definition file.
+   * Apply this knowledge to build reproducible computing environments.
+
+   In this demo, you will learn how to build your first container using Apptainer, starting from a simple definition file. Containers are crucial for creating reproducible, portable, and scalable environments that are isolated from the underlying infrastructure. This hands-on example will guide you through the process of defining and building a basic container, which is a foundational skill in using containers effectively in high-performance computing.
+
+.. prerequisites::
+
+   * No prerequisites for following the demo
+
+Building your first container is a significant milestone in adopting container technology for scientific computing. This demo is designed to provide you with the practical skills needed to create and manage these environments, which are critical for ensuring computational reproducibility and scalability across different HPC systems.
+
+In this tutorial, you will start with a basic container definition file that specifies the operating system, applications, and environment settings to be included in the container. You will then use Apptainer to build the container, making it ready for deployment and execution on an HPC cluster.
+
+    bash
+# Example definition file content for Apptainer
+Bootstrap: library
+From: alpine:latest
+
+%post
+    apk add --no-cache python3 py3-pip
+    pip3 install numpy
+
+%environment
+    export PATH=/usr/local/bin:$PATH
+    export LANG=C.UTF-8
+
+%runscript
+    echo "Container built from the definition file is running!"
+
+# Save the above content into a file named 'mydefinition.def'
+
+
+
+    bash
+# Build an Apptainer container from a definition file
+apptainer build mycontainer.sif mydefinition.def
+
+
+This command constructs a container named 'mycontainer.sif' from the definition file 'mydefinition.def'. The `.sif` file format is a portable container format used by Apptainer. This step is essential as it compiles all the necessary components and software specified in your definition file into a single executable container.
+
+Summary
+-------
+This lesson taught you how to build a basic container using Apptainer from a definition file. You've learned a critical skill in containerization, enabling you to create reproducible and portable environments for a wide range of scientific applications. This capability is foundational in leveraging the full potential of HPC resources while ensuring consistency across different computing environments.
+