buildtest-nersc

This repository contains tests for Cori and Perlmutter using the buildtest framework.

Useful Links

• CDASH: https://my.cdash.org/index.php?project=buildtest-nersc
• Upstream Repo: https://github.com/buildtesters/buildtest-nersc
• Github Mirror Repo: https://software.nersc.gov/NERSC/buildtest-nersc

Buildtest References

• Documentation: https://buildtest.readthedocs.io/en/devel/
• Schema Docs: https://buildtesters.github.io/buildtest/
• Slack Channel: https://hpcbuildtest.slack.com
• Getting Started: https://buildtest.readthedocs.io/en/devel/getting_started.html
• Writing Buildspecs: https://buildtest.readthedocs.io/en/devel/buildspec_tutorial.html
• Contributing Guide: https://buildtest.readthedocs.io/en/devel/contributing.html

Setup

To get started, please connect to NERSC system and clone this repo and buildtest:

git clone https://github.com/buildtesters/buildtest
git clone https://github.com/buildtesters/buildtest-nersc

You will need python 3.8 or higher to install buildtest, on Cori/Perlmutter this can be done by loading python module and create a conda environment as shown below.

module load python
conda create -n buildtest
conda activate buildtest

Now let's install buildtest, assuming you have cloned buildtest in $HOME directory source the setup script. For csh users you need to source setup.csh

source ~/buildtest/setup.sh

# csh users
source ~/buildtest/setup.csh

Next, navigate to buildtest-nersc directory and set environment BUILDTEST_CONFIGFILE to point to config.yml which is the configuration file for NERSC system.

cd buildtest-nersc
export BUILDTEST_CONFIGFILE=$(pwd)/config.yml

Make sure the configuration is valid, this can be done by running the following.

buildtest config validate

buildtest will validate the configuration file with the JSON schema and report if configuration is valid. If you see an error during validation, please report an issue and try to fix the issue if you can.

Please make sure you are using tip of devel branch of buildtest when writing tests. You should sync your local devel branch with upstream fork, for more details see contributing guide.

First time around you should discover all buildspecs this can be done via buildtest buildspec find. The command below will find and validate all buildspecs in the buildtest-nersc repo and load them in buildspec cache. Note that one needs to specify --root to specify location where all buildspecs are located. For more details see https://buildtest.readthedocs.io/en/devel/configuring_buildtest/overview.html#buildspec-roots

cd buildtest-nersc
buildtest buildspec find --root buildspecs --rebuild -q

The buildspecs are loaded in buildspec cache file (JSON) that is used by buildtest buildspec find for querying cache. Subsequent runs will read from cache. For more details see buildspec interface.

Building Tests

Note: All tests are written in YAML using .yml extension

To build tests use buildtest build command for example we build all tests in system directory as follows

buildtest build -b system/

You can specify multiple buildspecs either files or directory via -b option

buildtest build -b slurm/partition.yml -b slurmutils/

You can exclude a buildspec via -x option this behaves same way as -b option so you can specify a directory or filepath which could be absolute path, or relative path. This is useful when you want to run multiple tests grouped in directory but exclude a few.

buildtest build -b slurm -x slurm/sinfo.yml

buildtest can run tests via tags which can be useful when grouping tests, to see a list of available tags you can run: buildtest buildspec find --tags

For instance if you want to run all lustre tests you can run the following:

buildtest build --tags lustre

For more details on buildtest test please see the buildtest tutorial

Tags Breakdown

When you write buildspecs, please make sure you attach one or more tags to the test that way your test will get picked up during one of the CI checks. Shown below is a summary of tag description

• daily - this tag is used for running daily system checks using gitlab CI. Tests should run relatively quick
• system - this tag is used for classifying all system tests that may include: system configuration, servers, network, cray tests. This tag should be used
• slurm - this tag is used for slurm test that includes slurm utility check, slurm controller, etc... This tag shouldn't be used for job submission that is managed by jobs tag. The slurm tag tests should be short running test that use a Local Executor.
• jobs - this tag is used for testing slurm policies by submitting jobs to scheduler.
• compile - this tag is used for compilation of application (OpenMP, MPI, OpenACC, CUDA, upc, bupc, etc...)
• e4s - this tag is used for running tests for E4S stack via spack test or E4S Testsuite.
• module - this tag is used for testing module system
• benchmark - this tag is used for benchmark tests. This can be application benchmarks, mini-benchmarks, kernels, etc...

You can see breakdown of tags and buildspec summary with the following commands

buildtest buildspec summary
buildtest buildspec find --group-by-tags

Querying Tests

You can use buildtest report and buildtest inspect to query tests. The commands differ slightly and data is represented differently. The buildtest report command will show output in tabular form and only show some of the metadata, if you want to access the entire test record use buildtest inspect command which displays the content in JSON format. For more details on querying tests see https://buildtest.readthedocs.io/en/devel/gettingstarted/query_test_report.html

CI Setup

Tests are run on schedule basis with one schedule corresponding to one gitlab job in .gitlab-ci.yml. The scheduled pipelines are configured in https://software.nersc.gov/NERSC/buildtest-nersc/-/pipeline_schedules. Each schedule has a variable TESTNAME defined to control which pipeline is run since we have multiple gitlab jobs. In the .gitlab-ci.yml we make use of conditional rules using only.

The scheduled jobs are run at different intervals (1x/day, 1x/week, etc...) at different times of day to avoid overloading the system. The gitlab jobs will run jobs based on tags, alternately some tests may be defined by running all tests in a directory (buildtest build -b apps). If you want to add a new scheduled job, please define a new schedule with an appropriate time. The target branch should be devel and define a unique variable used to distinguish scheduled jobs. Next, create a job in .gitlab-ci.yml that references the scheduled job and define variable TESTNAME in the scheduled pipeline.

The gitlab runner configuration is stored in $HOME/.gitlab-runner including the jacamar configuration (jacamar.toml).

Integrations

This project has integration with Slack to notify CI builds to buildtest Slack at #buildtest-nersc workspace. The integrations can be found at https://software.nersc.gov/NERSC/buildtest-nersc/-/settings/integrations.

The gitlab project https://software.nersc.gov/NERSC/buildtest-nersc has setup Github Integration in https://software.nersc.gov/NERSC/buildtest-nersc/-/settings/repository which is used for pull-mirroring and running CI/CD jobs. The pull-mirroring is configured in https://software.nersc.gov/NERSC/buildtest-nersc/-/settings/repository under Mirroring Repositories.

CDASH

buildtest will push test results to CDASH server at https://my.cdash.org/index.php?project=buildtest-nersc using buildtest cdash upload command.

Contributing Guide

To contribute back you will want to make sure your buildspec is validated before you contribute back, this could be done by running test manually buildtest build or see if buildspec is valid via buildtest buildspec find. It would be good to run your test and make sure it is working as expected, you can view test detail using buildtest inspect name <testname> or buildtest inspect query <testname>.

If you want to contribute your tests, please see CONTRIBUTING.md

Submitting an Issue

Please submit all issues to https://github.com/buildtesters/buildtest-nersc/issues. When creating an issue, please see the labels and try to select one or more labels to categorize issue. Please use the following labels depending on the type of issue you are reporting

• Bug: When creating an issue related to a test bug
• new-test: An issue for adding a new test
• E4S-Testsuite: Issues related to E4S testsuite project
• spack: Issues related to spack test
• documentation: Issues with documentation such as README.md, CONTRIBUTING.md
• gitlab-ci: Issues with Gitlab CI/CD