Instructions for DiRAC HPCs

This page gives step-by-step instructions for the installation and running of Triumvirate in development mode on DiRAC HPCs including DIaL3 and COSMA8.

DIaL3

Clone the repository

In a clean login-shell session, change to the directory where you want keep the Triumvirate code repository. Clone and enter the repository:

git clone https://github.com/MikeSWang/Triumvirate.git
cd Triumvirate/

Load software modules

Load the necessary HPC software modules which may include the compiler/package manager and dependencies such as OpenMP, GSL and FFTW.

For DIaL3, the following is/are recommended:

module load gcc/13.2.0 openmpi/5.0.2
module load gsl/cray
module load fftw/3.3.10/rjkwzk-omp

If you use the Python interface of Triumvirate, you also need the Python/Conda module:

module load miniconda3/4.10.3

Set up the environment (Python only)

It is recommended that you install the Python package in a separate dedicated Conda environment:

conda create -n <your-env-name>
conda activate <your-env-name>

Build and install

In general, execute

make clean
make -j install useomp=true

to install an OpenMP-enabled version of Triumvirate for both the C++ executable and Python package.

Replace install above with cppinstall or pyinstall if you only need the C++ or Python interface respectively. See the official documentation for more details.

Checks and validation

To check the C++ executable is built correctly, run this in the shell from the repository root:

$ build/bin/triumvirate -V

     //\        ___  __                      __       ___  ___      
    //  \        |  |__) | |  | |\/| \  / | |__)  /\   |  |__    
   //    \       |  |  \ | \__/ |  |  \/  | |  \ /~~\  |  |___ 
  //      \                                                         
 //________\    • Three-Point Clustering Measurements in LSS •      

Copyright (C) 2023 Mike S Wang & Naonori S Sugiyama

LICENCE >

GPL-3.0-or-later

This program is free software: you can redistribute it and/or modify 
it under the terms of the GNU General Public License as published by 
the Free Software Foundation, either version 3 of the License, or    
(at your option) any later version.                                  
                                                                     
This program is distributed in the hope that it will be useful, but  
WITHOUT ANY WARRANTY; without even the implied warranty of           
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU     
General Public License for more details.                             
                                                                     
You should have received a copy of the GNU General Public License    
along with this program. If not, see <https://www.gnu.org/licenses/>.

PROGRAM INFORMATION >

Triumvirate version: 0.5.0-122-gafeebcab
GSL version: 2.7.1
FFTW version: fftw-3.3.10-sse2-avx-avx2-avx2_128
OpenMP version: 201511

which should print out a similar message to the one shown above.

To check the Python package is installed correctly, run this in the shell:

$ python -c "from triumvirate import validate_installation as vi; vi()"
Installation of Triumvirate has been validated.

which should print out the message above.

Future updates

Use git pull to pull future updates into your local repository, and rebuild Triumvirate as above.

Batch job submission

As an example (which may require modification for your own use case), here is a simplified batch script for a Python script that uses Triumvirate:

#!/usr/bin/env bash

#SBATCH --job-name=<your-job-name>
#SBATCH --account=<dpnnn>
#SBATCH --partition=slurm
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=128


# -- Set-up --------------------------------------------------------------

module load gcc/13.2.0 openmpi/5.0.2
module load gsl/cray
module load fftw/3.3.10/rjkwzk-omp
module load miniconda3/4.10.3

source activate <your-env-name>

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=threads
export OMP_PROC_BIND=spread


# -- Run -----------------------------------------------------------------

srun --unbuffered python <python-script>

COSMA8

The instructions for COSMA8 are nearly identical, but we recommend the following modules:

module load gnu_comp/14.1.0
module load openmpi/5.0.3
module load gsl/2.8
module load fftw/3.3.10
module load python/conda3-2023.09

Loading the Python/conda3-2023.09 may prompt you to run the following in the shell:

eval "$(/cosma/local/anaconda3/202309/bin/conda shell.bash hook)"

The other element that needs changing is the partition name in the batch script. See this COSMA8 webpage for what partitions you should use, although we recommend cosma8 by default if you set the number of CPUs to 128 in your job script.