Building on Skybridge with Intel

Building Trilinos/Albany on Skybridge with Intel requires third-party libraries Boost, HDF5, and NetCDF. To begin, we set up the modules:

module purge
module load intel/14.0
module load openmpi-intel/1.8

Building Boost

Boost versions >= 1.55 don't mix with Intel 14.0, so we use Boost v1.54.

mkdir $HOME/boost-1.54-intel
tar xzvf $HOME/boost_1_54_0.tar.gz
cd $HOME/boost_1_54_0
./bootstrap.sh --prefix=$HOME/boost-1.54-intel --with-toolset=intel-linux

Edit the project-config.jam text file to add the MPI line below:

if ! intel-linux in [ feature.values <toolset> ]
{
    using intel-linux ;
    using mpi : /opt/openmpi-1.8-intel/bin/mpicxx ;
}

Back in terminal, build and install with b2.

./b2 -j 8
./b2 install

Building HDF5

mkdir $HOME/hdf5-1.8.12-intel
tar xzvf $HOME/hdf5-1.8.12.tar.gz
cd $HOME/hdf5-1.8.12
./configure --prefix=$HOME/hdf5-1.8.12-intel --enable-parallel CC=mpicc CFLAGS=-fpic CXX=mpicxx CXXFLAGS=-fpic FC=mpifort FCFLAGS=-fpic
make -j 8
make install

If you find linking problems while building HDF5, try using the CMake GUI.

mkdir build
cd build
ccmake ..

Edit the individual lines to show the correct paths and options.

Building NetCDF

mkdir $HOME/netcdf-4.3.2-intel
tar xzvf $HOME/netcdf-4.3.2.tar.gz
cd $HOME/netcdf-4.3.2.tar.gz

Edit include/netcdf.h to change the following lines (near line 232):

#define NC_MAX_DIMS      65536    /* max dimensions per file */
#define NC_MAX_ATTRS      8192    
#define NC_MAX_VARS     524288    /* max variables per file */
#define NC_MAX_NAME        256 
#define NC_MAX_VAR_DIMS      8    /* max per variable dimensions */

Back in terminal,

./configure --prefix=$HOME/netcdf-4.3.2-intel --enable-netcdf-4 --disable-dap CC=mpicc CXX=mpicxx FC=mpifort CPPFLAGS="-I$HOME/hdf5-1.8.12-intel/include" CFLAGS=-fpic LDFLAGS="-L$HOME/hdf5-1.8.12-intel/lib" CXXFLAGS=-fpic FCFLAGS=-fpic
make -j 8
make install

Building Trilinos

Use the configure script provided by Albany:Building Trilinos. Save it in configureTrilinosForAlbany-kokkosSerial.sh. Note that SuperLU is not required. Edit the paths near the top of the configure script file to reflect the paths of your Boost, HDF5, and NetCDF libraries.

# Modify these paths for your system.
TRILINSTALLDIR=$HOME/trilinos-intel-kokkosSerial/install
BOOSTDIR=$HOME/boost-1.54-intel
NETCDFDIR=$HOME/netcdf-4.3.2-intel
HDF5DIR=$HOME/hdf5-1.8.12-intel

Back in terminal,

./configureTrilinosForAlbany-kokkosSerial.sh
make -j 8
make install 

Building Albany

cd $HOME/Albany
mkdir build
cd build

Modify the following configureAlbany.sh script to reflect how you will use Albany.

#!/bin/bash

rm -rf CMakeFiles/ CMakeCache.txt

BUILD_DIR=`pwd`

TRILINOS_INSTALL_DIR=$HOME/trilinos-intel-kokkosSerial/install

cmake \
-D ALBANY_TRILINOS_DIR=$TRILINOS_INSTALL_DIR \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-D CMAKE_CXX_FLAGS:STRING="-std=c++11" \
-D ENABLE_LCM:BOOL=OFF \
-D ENABLE_MOR:BOOL=OFF \
-D ENABLE_ALBANY_EPETRA_EXE:BOOL=OFF \
-D ENABLE_FELIX:BOOL=OFF \
-D ENABLE_HYDRIDE:BOOL=OFF \
-D ENABLE_SEE:BOOL=OFF \
-D ENABLE_AMP:BOOL=OFF \
-D ENABLE_SCOREC:BOOL=OFF \
-D ENABLE_QCAD:BOOL=OFF \
-D ENABLE_SG_MP:BOOL=OFF \
-D ENABLE_ASCR:BOOL=OFF \
-D ENABLE_AERAS:BOOL=ON \
-D ENABLE_64BIT_INT:BOOL=OFF \
-D ENABLE_INSTALL:BOOL=OFF \
-D CMAKE_INSTALL_PREFIX:PATH=$BUILD_DIR/install \
-D ENABLE_DEMO_PDES:BOOL=ON \
-D ENABLE_MPAS_INTERFACE:BOOL=OFF \
-D ENABLE_CISM_INTERFACE:BOOL=OFF \
..

Back in terminal,

./configureAlbany.sh

Building Albany requires a lot of memory. If you build with make -j 8 on the login node, you may require more memory than the login node allows, and the compiler may output a core dump error. If this happens, use fewer than 8 tasks for a parallel build or use a serial build.

make