Updated CI driving scripts. Take config out of the scripts that drive…

… the CI. Add config for aws and vega. Add documentation in README on how to do regular runs on one's own system, reusing this setup of scripts/config

CI/README.md

@@ -0,0 +1,43 @@
+# Setting up EESSI test suite CI
+
+To set up regular runs for the EESSI test suite on a system, four things are needed:
+
+1. The variable `EESSI_CI_SYSTEM_NAME` needs to be set in the environment
+2. A local checkout of the `CI` subdirectory of the EESSI test suite repository needs to be present
+3. The EESSI test suite repository needs to contain a file `CI/${EESSI_CI_SYSTEM_NAME}/ci_config.sh` with the configuration for the CI on that system
+4. Add running the `run_reframe_wrapper.sh` to your `crontab`
+
+## Setting the `EESSI_CI_SYSTEM_NAME`
+The easiest way to set this system name is to export it in the `.bashrc` of the login that will run the CI pipeline.
+
+## Checking out the CI folder from the EESSI test-suite
+You can clone the full EESSI test suite
+```
+git clone https://github.com/EESSI/test-suite.git
+```
+Or do a sparse checkout
+```
+git clone -n --depth=1 --filter=tree:0 https://github.com/EESSI/test-suite.git
+cd test-suite
+git sparse-checkout set --no-cone CI
+git checkout
+```
+
+## Creating a CI configuration file
+If you are adding CI on a new system, first, pick a name for that system (we'll refer to this as `EESSI_CI_SYSTEM_NAME`). Using the example in `CI/aws_citc/ci_config.sh`, you can adapt the config to your needs.
+It should define:
+- `TEMPDIR` (mandatory): the temporary directory in which the CI pipeline can check out repositories and install ReFrame.
+- `REFRAME_VERSION` (mandatory): the version of ReFrame you'd like to use to drive the EESSI test suite in the CI pipeline.
+- `EESSI_VERSION` (mandatory): the version of the EESSI software stack you would like to be loaded & tested in the CI pipeline.
+- `RFM_CONFIG_FILES` (mandatory): the location of the ReFrame configuration file to be used for this system.
+- `RFM_CHECK_SEARCH_PATH` (mandatory): the search path where ReFrame should search for tests to run in this CI pipeline.
+- `RFM_CHECK_SEARCH_RECURSIVE` (mandatory): whether ReFrame should search `RFM_CHECK_SEARCH_PATH` recursively.
+- `RFM_PREFIX`: the prefix in which ReFrame stores all the files.
+Note that the stdout and stderr of the CI pipeline is _not_ configurable and always ends up in `${HOME}/EESSI_CI_LOGS` (unless this is changed in `run_reframe_wrapper.sh`).
+
+## Add `run_reframe_wrapper.sh` to your `crontab`
+This line depends on how often you want to run the tests, and where the `run_reframe_wrapper.sh` is located exactly. Assuming you checked out the EESSI test suite repository in your home dir:
+```
+echo "0 0 * * SUN ${HOME}/test-suite/CI/run_reframe_wrapper.sh" | crontab -
+```
+Would create a cronjob running weekly on Sundays. See the crontab manual for other schedules.

CI/aws_citc/ci_config.sh

@@ -0,0 +1,13 @@
+# Configurable items
+TEMPDIR=$(mktemp --directory --tmpdir=/tmp  -t rfm.XXXXXXXXXX)
+TAGS="--tag CI --tag 1_node|2_nodes"
+# The hpctestlib as well as the reframe command will be based on this version
+# May be adapted later to simply use reframe from the EESSI software stack
+REFRAME_VERSION=4.3.2
+EESSI_VERSION=2023.06
+
+# ReFrame configuration
+export RFM_CONFIG_FILES="${TEMPDIR}/test-suite/config/aws_citc.py"
+export RFM_CHECK_SEARCH_PATH="${TEMPDIR}/test-suite/eessi/testsuite/tests/"
+export RFM_CHECK_SEARCH_RECURSIVE=1
+export RFM_PREFIX="${HOME}/reframe_CI_runs"

CI/izum_vega/ci_config.sh

@@ -0,0 +1,13 @@
+# Configurable items
+TEMPDIR=$(mktemp --directory --tmpdir=/tmp  -t rfm.XXXXXXXXXX)
+TAGS="--tag CI --tag 1_node|2_nodes"
+# The hpctestlib as well as the reframe command will be based on this version
+# May be adapted later to simply use reframe from the EESSI software stack
+REFRAME_VERSION=4.3.2
+EESSI_VERSION=2023.06
+
+# ReFrame configuration
+export RFM_CONFIG_FILES="${TEMPDIR}/test-suite/config/izum_vega.py"
+export RFM_CHECK_SEARCH_PATH="${TEMPDIR}/test-suite/eessi/testsuite/tests/"
+export RFM_CHECK_SEARCH_RECURSIVE=1
+export RFM_PREFIX="${HOME}/reframe_CI_runs"

CI/run_reframe.sh

@@ -0,0 +1,70 @@
+#!/bin/bash
+# Author: Caspar van Leeuwen
+# Description: This script can be used to do regular runs of the ReFrame test suite, e.g. from a cronjob.
+# Setup instructions: make sure you have your github access key configured in your .ssh/config
+# i.e. configure an entry with HostName github.com and IdentityFile pointing to the ssh key registered with Github
+
+# Get directory of the current script
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# Check if EESSI_CI_SYSTEM_NAME is defined
+if [ -z "${EESSI_CI_SYSTEM_NAME}" ]; then 
+    echo "You have to define the RFM_CI_SYSTEM_NAME environment variable in order to run the EESSI test suite CI" > /dev/stderr
+    exit 1
+fi
+
+# Check if CI_CONFIG file file exists
+CI_CONFIG="${SCRIPT_DIR}/${EESSI_CI_SYSTEM_NAME}/ci_config.sh"
+if [ ! -f ${CI_CONFIG} ]; then
+    echo "File ${CI_CONFIG} does not exist. Please check your RFM_CI_SYSTEM_NAME (${EESSI_CI_SYSTEM_NAME}) and make sure the directory in which the current script resides (${SCRIPT_DIR}) contains a subdirectory with that name, and a CI configuration file (ci_config.sh) inside". > /dev/stderr
+    exit 1
+fi
+
+# Set the CI configuration for this system
+source ${CI_CONFIG}
+
+# Create virtualenv for ReFrame using system python
+python3 -m venv ${TEMPDIR}/reframe_venv
+source ${TEMPDIR}/reframe_venv/bin/activate
+python3 -m pip install reframe-hpc==${REFRAME_VERSION}
+
+# Clone reframe repo to have the hpctestlib:
+git clone https://github.com/reframe-hpc/reframe.git --branch v${REFRAME_VERSION} ${TEMPDIR}/reframe
+export PYTHONPATH=${PYTHONPATH}:${TEMPDIR}/reframe
+
+# Clone test suite repo
+git clone https://github.com/EESSI/test-suite.git ${TEMPDIR}/test-suite
+export PYTHONPATH=${PYTHONPATH}:${TEMPDIR}/test-suite/
+
+# Start the EESSI environment
+unset MODULEPATH
+source /cvmfs/pilot.eessi-hpc.org/versions/${EESSI_VERSION}/init/bash
+
+# Needed in order to make sure the reframe from our TEMPDIR is first on the PATH,
+# prior to the one shipped with the 2021.12 compat layer
+# Probably no longer needed with newer compat layer that doesn't include ReFrame
+deactivate
+source ${TEMPDIR}/reframe_venv/bin/activate
+
+# Print ReFrame config
+echo "Starting CI run with the follwing settings:"
+echo "TMPDIR=${TMPDIR}"
+echo "PYTHONPATH: ${PYTHONPATH}"
+echo "TAGS=${TAGS}"
+echo "ReFrame executable=$(which reframe)"
+echo "ReFrame version=$(reframe --version)"
+echo "ReFrame config file=${RFM_CONFIG_FILES}"
+echo "ReFrame check search path=${RFM_CHECK_SEARCH_PATH}"
+echo "ReFrame check search recursive=${RFM_CHECK_SEARCH_RECURSIVE}"
+echo "ReFrame prefix=${RFM_PREFIX}"
+
+# List tests
+echo "Listing tests:"
+reframe ${TAGS} --list
+
+# Run
+echo "Run tests:"
+reframe ${TAGS} --run
+
+# Cleanup
+rm -rf ${TEMPDIR}

CI/run_reframe_wrapper.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+# Author: Caspar van Leeuwen
+# Description: wraps the run_reframe.sh script so that all stdout and stderr is easily be collected in a logfile
+# which has a datestamp in the name.
+
+# logfile
+LOGDIR=${HOME}/EESSI_CI_LOGS
+mkdir -p ${LOGDIR}
+
+datestamp=$(date +%Y%m%d_%H%M%S)
+LOGFILE=${LOGDIR}/rfm_${datestamp}.log
+touch $LOGFILE
+
+# Get directory of the current script
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# Execute run_reframe.sh, which should be in the same directory as the current script
+${SCRIPT_DIR}/run_reframe.sh > $LOGFILE 2>&1