Generate libEnsemble testing environments from templates for a variety of HPC platforms:
git clone https://github.com/Libensemble/libE-HPC-Templater.git cd libE-HPC-Templater; pip install -e .
Installs a templater
utility for generating tests:
$ templater --help Usage: templater [OPTIONS] COMMAND [ARGS]... libEnsemble Scaling Tests Templater Make test-specific adjustments to the .yaml files in libE- templater/platforms/PLATFORM/TEST Options: --help Show this message and exit. Commands: check Check a test directory (or `all`) for passes/fails. config Edit the base settings for a given machine. ls Display all currently supported machines and their tests make Make tests by machine and variant (or `all`). submit Try submitting each of the tests in a directory (or `all`).
See all supported machines and their tests:
$ templater ls Supported Machines and Tests: - Perlmutter: gpu_forces, forces, fbpic, - Swing: fbpic, - Summit: warpx, forces, - Bridges: warpx, forces, - Bebop: warpx, forces,
Modify machine-specific configuration:
$ templater config summit Summit Configuration: /Users/.../libE-templater/platforms/summit/platform.yaml ------------------------------------------------------------------------------------------------------ {'calling': {'nthreads': 4, 'sim_kill_minutes': 5, 'warpx_sim_app': "os.environ['HOME'] + " "'/warpx/Bin/main2d.gnu.TPROF.MPI.CUDA.ex'"}, 'submit': {'alloc_flags': 'smt1', 'conda_env_name': 'libe-gcc', 'job_name': 'libe_mproc', 'job_wallclock_minutes': 20, 'libe_wallclock': 15, 'project': 'csc314'}} Adjust any of the above platform parameters? (Y/N):
Create tests for a machine:
$ templater make theta all Writing: ./theta_warpx --test_mproc_MOM_zero_resource_workers_5w_8n --test_mproc_MOM_4w_8n --test_mproc_MOM_128w_256n Writing: ./theta_forces --test_mproc_MOM_4w_8n_fail_sim --test_mproc_MOM_4w_8n --test_MPI_balsam2_4w_4n --test_mproc_MOM_128w_128n --test_mproc_MOM_4w_8n_fail_submit
The platforms
directory contains:
- Platform-specific test configs and templates
- Platform-agnostic configs and templates in
all
.
Each platform directory has:
- Test-type specific directories
- Templates for scheduler submission scripts
- A platform.yaml
with universal parameters for that platform
Each test directory has:
- A stage
directory containing files to copy into the output test directory
- .yaml
files corresponding to test-variants
Any staged shell-scripts prefixed with "prepare" will be run by the templater in the output directory. This can help copy files into test-variant directories or adjust permissions on shell scripts.
Within all .yaml
files:
calling:
: parameters in libEnsemble python-initialization scriptssubmit:
: parameters in batch submission scripts
Note the following about platform.yaml
:
- Parameters don't have to be universal for all tests. For instance,
"nthreads": 1
can be included for each WarpX test, but doesn't have to appear in Forces templates.- Parameters can also appear in test-specific configurations. Test configurations will override values from
platform.yaml
.
Suppose we want to define a new test "particles"
, only for Theta, with mpi_128-nodes
and multiprocess_64-nodes
variants.
1) Place Jinja templates for calling scripts and submission scripts in platforms/all
or platforms/theta
.
- Create a test directory,
platforms/theta/particles
.
3) Place configuration .yaml
files to populate templates within this new directory.
In this case, they'll be named mpi_128-nodes.yaml
and multiprocess_64-nodes.yaml
.
They must contain "calling"
and "submit"
keys matching a "template"
key-value pairs and any number of other key-value pairs.
For example:
calling: sample_parameter: true template: my_calling_template.py submit: another_parameter: 123 template: my_submission_template.sh
4) (Optional) place files to copy over to the eventual output directory, theta_particles
,
within a new directory stage
inside the above test directory.