Building Albany using Spack

Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. With Spack, you can build a package with multiple versions, configurations, platforms, and compilers, and all of these builds can coexist on the same machine. The following links provide some documentation on Spack:

• Spack web page https://spack.io
• Spack documentation is at http://spack.readthedocs.io/en/latest/
• Spack Google Group https://groups.google.com/forum/#!forum/spack
• Specific examples of ~/.spack/linux/compilers.yaml files are highlighted below
• Some slides by Xylar Asay-Davis (@xylar) about the Albany-supported spack build

A working Albany build is supported in the E3SM fork of the spack repo: https://github.com/E3SM-Project/spack.

Clone and setup Spack from E3SM-Project repo

	git clone [email protected]:E3SM-Project/spack.git
        cd spack
        git checkout develop
	. share/spack/setup-env.sh

This approach requires uploading the ssh-keys for your machine to github (under Settings). An alternative to the first line above is: https://github.com/E3SM-Project/spack.git if you prefer cloning using HTTPS instead of SSH, which does not require uploading ssh-keys.

Note that spack will try to download and install packages from the web. If your machine is behind a proxy, you will need to set proxies (export http_proxy=..., export https_proxy=...) prior to running . share/spack/setup-env.sh above.

Building and installing Albany using SPACK

	spack --insecure install --dirty --keep-stage albany

The --keep-stage option will ensure that the Albany build is kept after the installation of Albany is complete. This is required if you plan to run the Albany regression tests post-build.

If you are interested in building Albany for the purpose of installing MPAS-Albany on top of Albany, please use the following command:

            spack --insecure install --dirty --keep-stage albany+mpas

Note that Albany requires one to use one of the following compilers:

            Clang      4.0.0 or higher
            GCC        5.3.0 or higher
            Intel     17.0.0 or higher
            NVCC      9.2.88 or higher

Instructions are provided below for how to check what compiler you have, how to build a new compiler using spack, and how to tell spack with what compiler to build Albany.

Determining the installed location of Albany

	spack location -i albany

Misc spack installation notes

• Building some of the Albany dependencies might result in certificate errors. To ignore certificates, use the command

    spack --insecure install albany
  

  Note that ignoring ssh certificate issues can compromise your system's security. Note that spack --insecure install albany is equivalent to spack -k install albany. The --insecure option is in general required on machines behind firewalls (with proxies).

• To build Albany your current shell environment, instead of the SPACK environment

    spack install --dirty albany
  

• To build Albany using 16 processors with verbose build output

    spack install -j 16 -v albany
  

• To run the Albany regression tests post-build, change into the build stage directory and run them with CTest

    spack cd albany
    cd ../spack-build
    ctest
  

• Uninstalling a spackage: spack unistall albany

Compilers.yaml Examples

cee-compute Machines at Sandia

The following is an example compilers.yaml file. It is typically found in the following location on Linux: ~/.spack/$arch/compilers.yaml.

	compilers:
	- compiler:
	    environment: {}
	    extra_rpaths: []
	    flags: {}
	    modules: []
	    operating_system: rhel6
	    paths:
	      cc: /sierra/sntools/SDK/compilers/clang/6.00-RHEL6/bin/clang
	      cxx: /sierra/sntools/SDK/compilers/clang/6.00-RHEL6/bin/clang++
	      f77: null
	      fc: null
	    spec: [email protected]
	    target: x86_64
	- compiler:
	    environment:
	      set:
	        LD_LIBRARY_PATH : /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/lib:/sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/lib64
	    extra_rpaths: []
	    flags: {}
	    modules: []
	    operating_system: rhel6
	    paths:
	      cc: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gcc
	      cxx: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/g++
	      f77: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gfortran
	      fc: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gfortran
	    spec: [email protected]
	    target: x86_64

To generate a default compilers.yaml file, please run spack compiler find.

Changing the compiler used with spack

To see what compilers you have on your machine, you can run spack compilers. Lets say this command returns the following:

==> Available compilers
-- gcc rhel7-x86_64 ---------------------------------------------
[email protected]  [email protected]  [email protected]  [email protected]

and we wish to build Albany for MPAS with gcc-7.2.0. This can be done as follows:

                 spack --insecure install --dirty --keep-stage albany%[email protected]+mpas

Lets say the compiler you wish to use is not available on your system, e.g., it is [email protected]. One first needs to build the compiler using spack:

                 spack --insecure install --dirty --keep-stage [email protected]

Once that is done, one needs to make a note of the location in which the compiler was installed, and update one's compilers.yaml file to have an entry for the new compiler. Next, one can run spack compilers again and verify that [email protected] shows up. Finally, to build Albany with gcc-8.4.0, one runs the following command:

                 spack --insecure install --dirty --keep-stage albany%[email protected]+mpas

If you wish to use a different pre-built compiler for the spack build, you can add an entry to it in the ~/.spack/$arch/compilers.yaml file, following the example above.

Changing the spack-stage directory

By default, spack will stage its builds in the /tmp/$USER/spack-stage directory. To change this, edit the spack/etc/spack/default/config.yaml, line 62 to specify the new path, e.g.,

                build_stage:
                   - /scratch/albany/nightlySpackBuild/spack-stage

Telling spack to use pre-built TPLs

In some cases, a user may not want spack to build every single TPL on which Albany depends; rather, one may want to have SPACK use pre-built TPLs. This can be accomplished by creating a ~/.spack/packages.yaml file that points to the pre-installed TPLs. This file will look as follows:

                packages:
                    openmpi:
                        paths:
                            [email protected]: /projects/sems/install/rhel7-x86_64/sems/v2/tpl/openmpi/1.10.7/gcc/10.1.0/base/7jgrwmo
                        buildable: False
                    cmake:
                        paths:
                            [email protected]: /projects/sems/install/rhel7-x86_64/sems/v2/utility/cmake/3.23.1/gcc/8.3.0/base/frlm6uo
                        buildable: False
                    boost:
                        paths:
                            [email protected]: /projects/sems/install/rhel7-x86_64/sems/v2/tpl/boost/1.74.0/gcc/10.1.0/base/ua5ge5q
                        buildable: False