From 57c88b54890930d0a3c51fceba2880842c900845 Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:41:11 +0200 Subject: [PATCH 1/9] Remove references to DockerHub automated builds Remove references to DockerHub automated builds, as these are not available anymore for free plans. --- _episodes/02-pulling-images.md | 9 +++----- _episodes/08-gitlab-ci.md | 42 +++++----------------------------- 2 files changed, 9 insertions(+), 42 deletions(-) diff --git a/_episodes/02-pulling-images.md b/_episodes/02-pulling-images.md index 5b29914..a63683f 100644 --- a/_episodes/02-pulling-images.md +++ b/_episodes/02-pulling-images.md @@ -20,12 +20,9 @@ keypoints: Much like how GitHub allows for web hosting and searching for code, the [Docker Hub][docker-hub] image registry allows the same for Docker images. -Hosting and building of images is [free for public repositories][docker-hub-billing] and +Hosting images is [free for public repositories][docker-hub-billing] and allows for downloading images as they are needed. -Additionally, through integrations with GitHub and Bitbucket, Docker Hub repositories can -be linked against Git repositories so that -[automated builds of Dockerfiles on Docker Hub][docker-hub-builds] will be triggered by -pushes to repositories. +Additionally, you can configure your GitLab or GitHub CI pipelines to automatically build and push docker images to Docker Hub. # Pulling Images @@ -123,7 +120,7 @@ debian buster 00bf7fdd8baf 5 weeks ago {: .challenge} [docker-hub]: https://hub.docker.com/ -[docker-hub-billing]: https://hub.docker.com/billing-plans/ +[docker-hub-billing]: https://www.docker.com/pricing/ [docker-hub-builds]: https://docs.docker.com/docker-hub/builds/ [docker-docs-pull]: https://docs.docker.com/engine/reference/commandline/pull/ [docker-docs-images]: https://docs.docker.com/engine/reference/commandline/images/ diff --git a/_episodes/08-gitlab-ci.md b/_episodes/08-gitlab-ci.md index 3b7574f..d58f934 100644 --- a/_episodes/08-gitlab-ci.md +++ b/_episodes/08-gitlab-ci.md @@ -107,7 +107,9 @@ As we've seen, all these components can be encoded in a Dockerfile. So the first Now, you can proceed with updating your `.gitlab-ci.yml` to actually build the container during the CI/CD pipeline and store it in the gitlab registry. You can later pull it from the gitlab registry just as you would any other container, but in this case using your CERN credentials. > ## Not from CERN? -> If you do not have a CERN computing account with access to [gitlab.cern.ch](https://[gitlab.cern.ch), then everything discussed here is also available on [gitlab.com](https://gitlab.com) offers CI/CD tools, including the docker builder. Furthermore, you can do the same with github + dockerhub as explained in the next subsection. +> If you do not have a CERN computing account with access to [gitlab.cern.ch](https://[gitlab.cern.ch), then everything discussed here is also available on [gitlab.com](https://gitlab.com), which offers CI/CD tools, including the docker builder. +> Furthermore, you can achieve the same with GitHub + Github Container Registry. +> To learn more about these methods, see the next subsections. {: .callout} Add the following lines at the end of the `.gitlab-ci.yml` file to build the image and save it to the docker registry. @@ -176,42 +178,10 @@ In the `script` of this job there are three components : If the job runs successfully, then in the same way as described for [gitlab.cern.ch](https://[gitlab.cern.ch) in the previous section, you will be able to find the `Container Registry` on the left hand icon menu of your gitlab.com web browser and navigate to the image that was pushed to the registry. Et voila, c'est fini, exactement comme au CERN! -### Alternative: Automatic image building with github + dockerhub +### Alternative: GitHub.com -If you don't have access to [gitlab.cern.ch](https://gitlab.cern.ch), you can still -automatically build a docker image every time you push to a repository with github and -dockerhub. - -1. Create a clone of the skim and the fitting repository on your private github. - You can use the - [GitHub Importer](https://docs.github.com/en/github/importing-your-projects-to-github/importing-a-repository-with-github-importer) - for this. It's up to you whether you want to make this repository public or private. - -2. Create a free account on [dockerhub](http://hub.docker.com/). -3. Once you confirmed your email, head to ``Settings`` > ``Linked Accounts`` - and connect your github account. -4. Go back to the home screen (click the dockerhub icon top left) and click ``Create Repository``. -5. Choose a name of your liking, then click on the github icon in the ``Build settings``. - Select your account name as organization and select your repository. -6. Click on the ``+`` next to ``Build rules``. The default one does fine -7. Click ``Create & Build``. - -That's it! Back on the home screen your repository should appear. Click on it and select the -``Builds`` tab to watch your image getting build (it probably will take a couple of minutes -before this starts). If something goes wrong check the logs. - -DockerHub - -Once the build is completed, you can pull your image in the usual way. - -~~~bash -# If you made your docker repository private, you first need to login, -# else you can skip the following line -docker login -# Now pull -docker pull /: -~~~ -{: .source} +You can also build Docker images on [github.com](https://github.com) and push them to the GitHub Container Registry ([ghcr.io](https://ghcr.io)) with the help of [GitHub Actions](https://github.com/features/actions). +The bonus episode [Building and deploying a Docker container to Github Packages](/hsf-training-docker/12-bonus/index.html) explains how to do so. ## An updated version of `skim.sh` From 0de174cb39169bc41997fc2770b0e046de9b0834 Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:48:48 +0200 Subject: [PATCH 2/9] Update description of docker images --- _episodes/03-running-containers.md | 7 ++++--- _episodes/05-dockerfiles.md | 10 +++++----- _episodes/08-gitlab-ci.md | 4 ++-- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/_episodes/03-running-containers.md b/_episodes/03-running-containers.md index 218157c..51cb054 100644 --- a/_episodes/03-running-containers.md +++ b/_episodes/03-running-containers.md @@ -65,10 +65,11 @@ cat /etc/os-release {: .source} ~~~ -PRETTY_NAME="Debian GNU/Linux 9 (stretch)" +PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" -VERSION_ID="9" -VERSION="9 (stretch)" +VERSION_ID="11" +VERSION="11 (bullseye)" +VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" diff --git a/_episodes/05-dockerfiles.md b/_episodes/05-dockerfiles.md index 1525675..72be4c9 100644 --- a/_episodes/05-dockerfiles.md +++ b/_episodes/05-dockerfiles.md @@ -21,7 +21,7 @@ Docker images are built through the Docker engine by reading the instructions fr These text based documents provide the instructions through an API similar to the Linux operating system commands to execute commands during the build. The [`Dockerfile` for the example image][example-Dockerfile] being used is an example of -some simple extensions of the [official Python 3.6.8 Docker image][python-docker-image]. +some simple extensions of the [official Python 3.9 Docker image][python-docker-image] based on Debian Bullseye (`python:3.9-bullseye`). As a very simple example of extending the example image into a new image create a `Dockerfile` on your local machine @@ -113,8 +113,8 @@ python3 -c "import sklearn as sk; print(sk)" ||----w | || || -scikit-learn 0.21.3 - +scikit-learn 1.3.1 + ~~~ {: .output} @@ -241,9 +241,9 @@ way to bring them into the Docker build. [docker-docs-builder]: https://docs.docker.com/engine/reference/builder/ -[example-Dockerfile]: https://github.com/matthewfeickert/Intro-to-Docker/blob/master/Dockerfile +[example-Dockerfile]: https://github.com/matthewfeickert/intro-to-docker/blob/feat/update-2021-bootcamp/docker/Dockerfile [python-docker-image]: https://hub.docker.com/_/python -[cowsay]: https://packages.debian.org/jessie/cowsay +[cowsay]: https://packages.debian.org/bullseye/cowsay [scikit-learn]: https://scikit-learn.org [docker-docs-build]: https://docs.docker.com/engine/reference/commandline/build/ [docker-docs-ARG]: https://docs.docker.com/engine/reference/builder/#arg diff --git a/_episodes/08-gitlab-ci.md b/_episodes/08-gitlab-ci.md index d58f934..ca69bff 100644 --- a/_episodes/08-gitlab-ci.md +++ b/_episodes/08-gitlab-ci.md @@ -31,8 +31,8 @@ The goal of automated environment preservation is to create a docker image that As we've seen, all these components can be encoded in a Dockerfile. So the first step to set up automated image building is to add a Dockerfile to the repo specifying these components. > ## The `rootproject/root` docker image -> In this tutorial, we build our analysis environments on top of the `rootproject/root` base image ([link to project area on docker hub](https://hub.docker.com/r/rootproject/root)) with conda. This image comes with root 6.22 and python 3.7 pre-installed. It also comes with XrootD for downloading files from eos. -> The `rootproject/root` is itself built with a [Dockerfile](https://github.com/root-project/root-docker/blob/6.22.06-conda/conda/Dockerfile), which uses conda to install root and python on top of another base image (`continuumio/miniconda3`). +> In this tutorial, we build our analysis environments on top of the `rootproject/root` base image ([link to project area on docker hub](https://hub.docker.com/r/rootproject/root)) with conda. This image comes with root 6.22 and python 3.8 pre-installed. It also comes with XrootD for downloading files from eos. +> The `rootproject/root` is itself built with a [Dockerfile](https://github.com/root-project/root-docker/blob/6.22.06-conda/conda/Dockerfile), which uses conda to install root and python on top of another base image (`condaforge/miniforge3`). {: .callout} > ## Exercise (15 min) From 1d88a4e3cd263224ef680be247062d34d46f0c98 Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:50:15 +0200 Subject: [PATCH 3/9] Add note about SELinux --- _episodes/04-file-io.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/_episodes/04-file-io.md b/_episodes/04-file-io.md index 5b89edc..f10c81a 100644 --- a/_episodes/04-file-io.md +++ b/_episodes/04-file-io.md @@ -124,7 +124,13 @@ ls *.txt {: .source} >## Permission issues ->If you run into a `Permission denied` error, you can use ``sudo`` for a quick fix, if you have superuser access (but this is not recommended). +>If you are using Linux with SELinux enabled, you might run into a `Permission denied` error. +>Note that SELinux is enabled if the output of the command `getenforce status` is `Enforcing`. +>To fix the permission issue, append `:z` (lowercase!) at the end of the mount option, like this: +>```bash +>docker run --rm -it -v $PWD:/home/docker/data:z ... +>``` +>If this still does not fix the issue you can disable SELinux by running `sudo setenforce 0`, or you can try using `sudo` to execute docker commands, but both of these methods are not recommended. {: .callout} ~~~ From cb9fc896f891578f66e5d3eb99abf6d8510e87e9 Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:51:32 +0200 Subject: [PATCH 4/9] Use Kaniko to build docker images on CERN GitLab --- _episodes/08-gitlab-ci.md | 39 +++++++++++++++++--------- _episodes/09-containerized-analysis.md | 15 +++++++--- 2 files changed, 36 insertions(+), 18 deletions(-) diff --git a/_episodes/08-gitlab-ci.md b/_episodes/08-gitlab-ci.md index ca69bff..7b32aa5 100644 --- a/_episodes/08-gitlab-ci.md +++ b/_episodes/08-gitlab-ci.md @@ -107,22 +107,30 @@ As we've seen, all these components can be encoded in a Dockerfile. So the first Now, you can proceed with updating your `.gitlab-ci.yml` to actually build the container during the CI/CD pipeline and store it in the gitlab registry. You can later pull it from the gitlab registry just as you would any other container, but in this case using your CERN credentials. > ## Not from CERN? -> If you do not have a CERN computing account with access to [gitlab.cern.ch](https://[gitlab.cern.ch), then everything discussed here is also available on [gitlab.com](https://gitlab.com), which offers CI/CD tools, including the docker builder. +> If you do not have a CERN computing account with access to [gitlab.cern.ch](https://gitlab.cern.ch), then everything discussed here is also available on [gitlab.com](https://gitlab.com), which offers CI/CD tools, including the docker builder. > Furthermore, you can achieve the same with GitHub + Github Container Registry. > To learn more about these methods, see the next subsections. {: .callout} -Add the following lines at the end of the `.gitlab-ci.yml` file to build the image and save it to the docker registry. +Add the following lines at the end of the `.gitlab-ci.yml` file to build the image with Kaniko and save it to the docker registry. +For more details about building docker images on CERN's GitLab, see the [Building docker images](https://gitlab.docs.cern.ch/docs/Build%20your%20application/Packages%20&%20Registries/using-gitlab-container-registry#building-docker-images) docs page. ~~~yaml build_image: stage: build variables: - TO: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA - tags: - - docker-image-build + IMAGE_DESTINATION: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA + image: + # The kaniko debug image is recommended because it has a shell, and a shell is required for an image to be used with GitLab CI/CD. + name: gcr.io/kaniko-project/executor:debug + entrypoint: [""] script: - - ignore + # Prepare Kaniko configuration file + - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json + # Build and push the image from the Dockerfile at the root of the project. + - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $IMAGE_DESTINATION + # Print the full registry path of the pushed image + - echo "Image pushed successfully to ${IMAGE_DESTINATION}" ~~~ {: .source} @@ -143,7 +151,7 @@ You can also go to the container registry on the gitlab UI to see all the images Notice that the script to run is just a dummy 'ignore' command. This is because using the docker-image-build tag, the jobs always land on special runners that are managed by CERN IT which run a custom script in the background. You can safely ignore the details. > ## Recommended Tag Structure -> You'll notice the environment variable `TO` in the `.gitlab-ci.yml` script above. This controls the name of the Docker image that is produced in the CI step. Here, the image name will be `:-`. The shortened 8-character commit SHA ensures that each image created from a different commit will be unique, and you can easily go back and find images from previous commits for debugging, etc. +> You'll notice the environment variable `IMAGE_DESTINATION` in the `.gitlab-ci.yml` script above. This controls the name of the Docker image that is produced in the CI step. Here, the image name will be `:-`. The shortened 8-character commit SHA ensures that each image created from a different commit will be unique, and you can easily go back and find images from previous commits for debugging, etc. > > As you'll see tomorrow, it's recommended when using your images as part of a REANA workflow to make a unique image for each gitlab commit, because REANA will only attempt to update an image that it's already pulled if it sees that there's a new tag associated with the image. > @@ -152,31 +160,34 @@ Notice that the script to run is just a dummy 'ignore' command. This is because ### Alternative: GitLab.com -This training module is rather CERN-centric and assumes you have a CERN computing account with access to [gitlab.cern.ch](https://[gitlab.cern.ch). If this is not the case, then as with the [CICD training module](https://hsf-training.github.io/hsf-training-cicd/), everything can be carried out using [gitlab.com](https://gitlab.com) with a few slight modifications. These changes are largely surrounding the syntax and the concept remains that you will have to specify that your pipeline job that builds the image is executed on a special type of runner with the appropriate `services`. However, unlike at CERN, there is not pre-defined `script` that runs on these runners and pushes to your registry, so you will have to write this script yourself but this will be little more than adding commands that you have been exposed to in previous section of this training like `docker build`. +This training module is rather CERN-centric and assumes you have a CERN computing account with access to [gitlab.cern.ch](https://gitlab.cern.ch). If this is not the case, then as with the [CICD training module](https://hsf-training.github.io/hsf-training-cicd/), everything can be carried out using [gitlab.com](https://gitlab.com) with a few slight modifications. +In particular, you will have to specify that your pipeline job that builds the image is executed on a special type of runner with the appropriate `services`. However, unlike at CERN, you can use the docker commands that you have seen in the previous episodes to build and push the docker images. Add the following lines at the end of the `.gitlab-ci.yml` file to build the image and save it to the docker registry. ~~~yaml -build image: +build_image: stage: build image: docker:latest services: - docker:dind + variables: + IMAGE_DESTINATION: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA script: - - docker build -t registry.gitlab.com/burakh/docker-training . + - docker build -t $IMAGE_DESTINATION . - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY - - docker push registry.gitlab.com/burakh/docker-training + - docker push $IMAGE_DESTINATION ~~~ {: .source} -In this job, the specific `image: docker:latest`, along with specifying the `services` to contain `docker:dind` is equivalent to the requesting the `docker-build-image` tag on [gitlab.cern.ch](https://[gitlab.cern.ch). If you are curious to read about this in detail, refer to the [official gitlab documentation](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html) or (this example)[https://gitlab.com/gitlab-examples/docker]. +In this job, the specific `image: docker:latest`, along with specifying the `services` to contain `docker:dind` are needed to be able to execute docker commands. If you are curious to read about this in detail, refer to the [official gitlab documentation](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html) or [this example](https://gitlab.com/gitlab-examples/docker). In the `script` of this job there are three components : - - [`docker build`](https://docs.docker.com/engine/reference/commandline/build/) : This is performing the same build of our docker image to the tagged image which we will call `registry.gitlab.com/burakh/docker-training` + - [`docker build`](https://docs.docker.com/engine/reference/commandline/build/) : This is performing the same build of our docker image to the tagged image which we will call `:-` - [`docker login`](https://docs.docker.com/engine/reference/commandline/login/) : This call is performing [an authentication of the user to the gitlab registry](https://docs.gitlab.com/ee/user/packages/container_registry/#authenticating-to-the-gitlab-container-registry) using a set of [predefined environment variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) that are automatically available in any gitlab repository. - [`docker push`](https://docs.docker.com/engine/reference/commandline/push/) : This call is pushing the docker image which exists locally on the runner to the gitlab.com registry associated with the repository against which we have performed the authentication in the previous step. -If the job runs successfully, then in the same way as described for [gitlab.cern.ch](https://[gitlab.cern.ch) in the previous section, you will be able to find the `Container Registry` on the left hand icon menu of your gitlab.com web browser and navigate to the image that was pushed to the registry. Et voila, c'est fini, exactement comme au CERN! +If the job runs successfully, then in the same way as described for [gitlab.cern.ch](https://gitlab.cern.ch) in the previous section, you will be able to find the `Container Registry` on the left hand icon menu of your gitlab.com web browser and navigate to the image that was pushed to the registry. Et voila, c'est fini, exactement comme au CERN! ### Alternative: GitHub.com diff --git a/_episodes/09-containerized-analysis.md b/_episodes/09-containerized-analysis.md index 7e55b6b..d9856a3 100755 --- a/_episodes/09-containerized-analysis.md +++ b/_episodes/09-containerized-analysis.md @@ -52,11 +52,18 @@ To bring it all together, we can also preserve our fitting framework in its own > > build_image: > > stage: build > > variables: -> > TO: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA -> > tags: -> > - docker-image-build +> > IMAGE_DESTINATION: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA +> > image: +> > # The kaniko debug image is recommended because it has a shell, and a shell is required for an image to be used with GitLab CI/CD. +> > name: gcr.io/kaniko-project/executor:debug +> > entrypoint: [""] > > script: -> > - ignore +> > # Prepare Kaniko configuration file +> > - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json +> > # Build and push the image from the Dockerfile at the root of the project. +> > - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $IMAGE_DESTINATION +> > # Print the full registry path of the pushed image +> > - echo "Image pushed successfully to ${IMAGE_DESTINATION}" > > > > [... rest of .gitlab-ci.yml] > > ~~~ From 57b973d55d67b4d33545d9b79ff5be78061f13ce Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:53:13 +0200 Subject: [PATCH 5/9] Improve bonus episode on GitHub packages Fix the rendering of the YAML config file, as `{{ ... }}` were considered as part of the template and would be hidden. Build and publish the docker image only on push and not for each pull request. Update the GitHub actions to the latest versions. --- _episodes/12-bonus.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/_episodes/12-bonus.md b/_episodes/12-bonus.md index fc7aa90..ab2b31a 100644 --- a/_episodes/12-bonus.md +++ b/_episodes/12-bonus.md @@ -8,8 +8,8 @@ questions: objectives: - To be able to build a Docker container and share it via GitHub packages keypoints: --  Python packages can be installed in Docker images along with ubuntu packages. --  It is possible to publish and share Docker images over github packages. +- Python packages can be installed in Docker images along with ubuntu packages. +- It is possible to publish and share Docker images over github packages. --- > ## Prerequisites @@ -22,7 +22,7 @@ keypoints: Python packages can be installed using a Docker image. The following example illustrates how to write a Dockerfile for building an image containing python packages. -```text +```docker FROM ubuntu:20.04 ENV DEBIAN_FRONTEND=noninteractive @@ -60,13 +60,15 @@ To do so, one needs to use GitHub CI/CD. A step-by-step guide is presented here. * **Step 4**: Copy-paste the content above and add it to the Dockerfile. (In principle it is possible to build this image locally, but we will not do that here, as we wish to build it with GitHub CI/CD). * **Step 5**: In the `Docker-build-deploy.yml` file, add the following content: -```text +{% raw %} +```yaml name: Create and publish a Docker image on: - pull_request: push: - branches: master + branches: + - main + - master env: REGISTRY: ghcr.io @@ -75,16 +77,17 @@ env: jobs: build-and-push-image: runs-on: ubuntu-latest + permissions: contents: read packages: write steps: - name: Checkout repository - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Log in to the Container registry - uses: docker/login-action@v2 + uses: docker/login-action@v3 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} @@ -92,18 +95,20 @@ jobs: - name: Docker Metadata id: meta - uses: docker/metadata-action@v4 + uses: docker/metadata-action@v5 with: images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} - name: Build and push Docker image - uses: docker/build-push-action@v3 + uses: docker/build-push-action@v5 with: context: . push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} ``` +{% endraw %} + The above script is designed to build and publish a Docker image with [GitHub packages](https://github.com/features/packages). From 8fb0f9f3405beef0e1a61e0905b988ce61202f6f Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:56:13 +0200 Subject: [PATCH 6/9] Fix broken links --- _episodes/01-introduction.md | 3 +-- _episodes/09-containerized-analysis.md | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/_episodes/01-introduction.md b/_episodes/01-introduction.md index 9bfbd32..3947565 100644 --- a/_episodes/01-introduction.md +++ b/_episodes/01-introduction.md @@ -43,8 +43,7 @@ very lightweight and fast to spin up to run. > > Singularity in particular is used widely in HPC, and particularly by CMS, so you may have need to familiarize yourself with it at some point. > -> The [RECAST FAQ](https://recast-docs.web.cern.ch/recast-docs/faq/#q-how-are-docker-and-singularity-different) includes a brief intro to the important differences between docker and singularity. -> To learn more about singularity, see for example the [HSF Training Singularity Module](https://github.com/hsf-training/hsf-training-singularity-webpage). +> To learn more about singularity, see for example the [HSF Training Singularity Module](https://hsf-training.github.io/hsf-training-singularity-webpage/). {: .callout} [docker-tutorial]: https://docs.docker.com/get-started diff --git a/_episodes/09-containerized-analysis.md b/_episodes/09-containerized-analysis.md index d9856a3..36a19f1 100755 --- a/_episodes/09-containerized-analysis.md +++ b/_episodes/09-containerized-analysis.md @@ -95,12 +95,12 @@ Now that we've preserved our full analysis environment in docker images, let's t > mkdir fitting_output > ~~~ > -> Find a partner and pull the image they've built for their skimming repo from the gitlab registry. Launch a container using your partner's image. Try to run the analysis code to produce the `histogram.root` file that will get input to the fitting repo, using the `skim_prebuilt.sh` script we created in the previous lesson for the first skimming step. You can follow the skimming instructions in [step 1](https://gitlab.cern.ch/awesome-workshop/awesome-analysis-eventselection-stage2/blob/master/README.md#step-1-skimming) and [step 2](https://gitlab.cern.ch/awesome-workshop/awesome-analysis-eventselection-stage2/blob/master/README.md#step-2-histograms) of the README. +> Find a partner and pull the image they've built for their skimming repo from the gitlab registry. Launch a container using your partner's image. Try to run the analysis code to produce the `histogram.root` file that will get input to the fitting repo, using the `skim_prebuilt.sh` script we created in the previous lesson for the first skimming step. You can follow the skimming instructions in [step 1](https://github.com/hsf-training/hsf-training-cms-analysis-snapshot#step-1-skimming) and [step 2](https://github.com/hsf-training/hsf-training-cms-analysis-snapshot#step-2-histograms) of the README. > > **Note:** We'll need to pass the output from the skimming stage to the fitting stage. To enable this, you can volume mount the `skimming_output` directory into the container. Then, as long as you save the skimming output to the volume-mounted location in the container, it will also be available locally under `skimming_output`. > > ### Part 2: Fitting -> Now, pull your partner's fitting image and use it to produce the final fit result. Remember to volume-mount the `skimming_output` and `fitting_output` so the container has access to both. At the end, the `fitting_output` directory on your local machine should contain the final fit results. You can follow the instructions in [step 4](https://gitlab.cern.ch/awesome-workshop/awesome-analysis-eventselection-stage2/blob/master/README.md#step-4-fit) of the README. +> Now, pull your partner's fitting image and use it to produce the final fit result. Remember to volume-mount the `skimming_output` and `fitting_output` so the container has access to both. At the end, the `fitting_output` directory on your local machine should contain the final fit results. You can follow the instructions in [step 4](https://github.com/hsf-training/hsf-training-cms-analysis-snapshot#step-4-fit) of the README. > > > ## Solution > > ### Part 1: Skimming From 55c9abf5748a491fdfc781617fe8189c3ae4e7f1 Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:56:43 +0200 Subject: [PATCH 7/9] Add hint on how to find compilation command --- _episodes/08-gitlab-ci.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_episodes/08-gitlab-ci.md b/_episodes/08-gitlab-ci.md index 7b32aa5..5b02a18 100644 --- a/_episodes/08-gitlab-ci.md +++ b/_episodes/08-gitlab-ci.md @@ -64,6 +64,7 @@ As we've seen, all these components can be encoded in a Dockerfile. So the first > ~~~ > {: .source} > +> Hint: have a look at `skim.sh` if you are unsure about how to complete the last `RUN` statement! > > ## Solution > > ~~~yaml > > # Start from the rootproject/root base image with conda From e04d978cec323095581bcbf11f369c781341fd05 Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Thu, 5 Oct 2023 15:57:19 +0200 Subject: [PATCH 8/9] Fix command to show current shell --- _episodes/11-entry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_episodes/11-entry.md b/_episodes/11-entry.md index 5e6f427..5e65bc7 100644 --- a/_episodes/11-entry.md +++ b/_episodes/11-entry.md @@ -30,7 +30,7 @@ docker run --rm -it python:3.7-slim /bin/bash Running this dumps us into a Bash session ~~~bash -printenv | grep SHELL +echo $SHELL ~~~ {: .source} From c916cf4c2a338680bc61c63e95f03a7da27247d8 Mon Sep 17 00:00:00 2001 From: Marco Donadoni Date: Sun, 15 Oct 2023 11:25:17 +0200 Subject: [PATCH 9/9] Update links to fitting dataset --- setup.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/setup.md b/setup.md index 338489e..eff8522 100644 --- a/setup.md +++ b/setup.md @@ -49,10 +49,14 @@ A few things to keep in mind if starting from these 'starter repos': * Regarding authentication with ``kinit``: * If you are from CERN and use gitlab.cern.ch: Remember to add your CERN credentials as CI/CD variables to both repos for the `kinit` authentication in the `.gitlab-ci.yml` files to work. - To do so, go to _Settings_ -> _CI/CD_ -> _Variables_ and create two new variables: `CERN_USER` should contain your CERN username, and `SERVICE_PASS` should contain your password. - * Else, you can remove the ``kinit`` line from `.gitlab-ci.yml` and use the public EOS dataset available at `root://eospublic.cern.ch//eos/root-eos/HiggsTauTauReduced`. -* For the fitting code repo, the [fit_simple](https://github.com/hsf-training/hsf-training-cms-analysis-snapshot-stats/blob/master/.gitlab-ci.yml#L5) step in `.gitlab-ci.yml` expects to receive the file `histograms.root` produced by the skimming code. In case you haven't had a chance to produce this file yet, it can be downloaded from [here](https://cernbox.cern.ch/index.php/s/LADW94G9fjY7hjF). - * If you are from CERN, you'll need to copy this file to your personal eos user space (`root://eosuser.cern.ch//eos/user/[first_letter_of_username]/[username]`) - * else: use the public eos link. + To do so, go to _Settings_ -> _CI/CD_ -> _Variables_ and create two new variables: + * `CERN_USER` should contain your CERN username + * `SERVICE_PASS` should contain your password. + * Else, you can remove the ``kinit`` line from `.gitlab-ci.yml` and use the public EOS datasets: + * `root://eospublic.cern.ch//eos/root-eos/HiggsTauTauReduced` for the skimming repo. + * `root://eospublic.cern.ch//eos/opendata/cms/upload/apb2023/histograms.root` for the fitting repo. +* For the fitting code repo, the [fit_simple](https://github.com/hsf-training/hsf-training-cms-analysis-snapshot-stats/blob/master/.gitlab-ci.yml#L5) step in `.gitlab-ci.yml` expects to receive the file `histograms.root` produced by the skimming code. In case you haven't had a chance to produce this file yet, it can be downloaded from [here](https://eospublichttp.cern.ch//eos/opendata/cms/upload/apb2023/histograms.root). In any case you can: + * use the public EOS datasets mentioned above. + * If you are from CERN, you can copy the downloaded file to your personal eos user space (`root://eosuser.cern.ch//eos/user/[first_letter_of_username]/[username]`). {% include links.md %}