Generic Instructions for HPC

Spack:

1. Follow getting started guide
   • Ensure develop branch is installed
   • Ensure desired compilers are known to spack, should be picked up by spack compiler find
   • Ensure system packages (that you don't want spack to install) are known to spack, some may be picked up by spack external find, others may require manual intervention.
2. You may want to specify the system MPI as the correct MPI for spack to use. This appears to be quite difficult, a starting template for the ~/.spack/packages.yamlmight be:

   packages:
     mpich:
     externals:
     - spec: [email protected]
       prefix: /opt/mpich
     buildable: False

   If Spack's PETSc package doesn't think this spec line is suitable it will try to build its own MPICH.

Make sure the spack setup-env.sh file has been sourced and you can run spack commands before moving to the next section.

Firedrake-repo

1. Clone the above repo, using

git clone https://github.com/firedrakeproject/firedrake-spack.git
# or
git clone [email protected]:firedrakeproject/firedrake-spack.git

1. Add the repository to spack spack repo add <repo directory>
2. Create an spack environment spack env create -d ./firedrake
3. Activate that environment spack env activate -p ./firedrake
   • To avoid a bunch of errors add a whole bunch of packages to the development package list:

   spack develop py-firedrake@develop
   spack develop libsupermesh@develop
   spack develop petsc@develop
   spack develop chaco@petsc
   spack develop py-fiat@develop
   spack develop py-finat@develop
   spack develop py-islpy@develop
   spack develop py-petsc4py@develop
   spack develop py-pyadjoint@develop
   spack develop py-pyop2@develop
   spack develop py-coffee@develop
   spack develop py-loopy@develop
   spack develop py-cgen@develop
   
   spack develop py-codepy@develop
   spack develop py-genpy@develop
   spack develop py-tsfc@develop
   spack develop py-ufl@develop
   
4. Install firedrake using spack add py-firedrake@develop %gcc ^mpich ^openblas
   • %gcc specifies the compiler, you may wish to specify a version (eg: %[email protected]), ommiting this will use the default compiler
   • ^mpich specifies which MPI to use. If you set an MPI system package be sure to use this (eg: ^[email protected])
   • ^openblas specifies which BLAS/LAPACK library to use. If you set an BLAS/LAPACK system package be sure to use this (eg: ^[email protected])
   • You can further specify the Python version by adding ^[email protected] for Python 3.10 or setting the system python if it's set up in system packages
5. spack install Will install everything
   • Add --fail-fast to stop at the first package with an install error
   • Adding 2>&1 | tee spack-firedrake-install.log to the end of the command will save all output to a log file that you can send if you need someone else to look over the output.
   • spack install --fail-fast 2>&1 | tee spack-firedrake-install.log
6. Test you can import Firedrake by running python -c "from firedrake import *"
   • If this fails, before trying anything else, deactivate the environment with spack env deactivate and reactivate with spack env activate -p ./firedrake (as above) and try running python -c "from firedrake import *" again. This appears to be a shortcoming of spack (related to#10801?).
7. Run the basic functionality tests:

   cd $SPACK_ENV/py-firedrake
   pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"

8. Run the full test suite:

   cd $SPACK_ENV/src/firedrake
   pytest tests