Rhel6 using Intel

This set of build instructions are for the Albany suite on Rhel6 linux.

The compilers are Intel 17.0, Intel MPI 5.1, and RHEL 6 linux.

Modules on CEE:

	module load sierra-devel/intel-17.0.1-intelmpi-5.1
	module load sierra-mkl/17.0-2017.2.174

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

• The TPLs are installed in /projects/albany

Building ZLib

• Version 1.2.11

    CC=mpiicc CFLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" ./configure --64 --prefix=/projects/albany/intel5.1
    make -j 4
    make install
  

Building Yaml

• git clone [email protected]:jbeder/yaml-cpp.git

    INTEL_MPI_DIR=/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64
  
    cmake \
    	-DCMAKE_CXX_COMPILER:STRING=${INTEL_MPI_DIR}/bin/mpiicpc \
    	-DCMAKE_CXX_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \
    	-DCMAKE_C_COMPILER:STRING=${INTEL_MPI_DIR}/bin/mpiicc \
    	-DCMAKE_C_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \
    	-DCMAKE_INSTALL_PREFIX:PATH=/projects/albany/intel5.1 \
    	-DCMAKE_BUILD_TYPE:STRING=Release \
    	-DBUILD_SHARED_LIBS:BOOL=ON \
    	-DYAML_CPP_BUILD_TOOLS:BOOL=OFF \
    ..
  
    make -j 4
    make install
  

Building HDF5

• Version 1.10.0 patch 1

    ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
     CFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
     FCFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" --enable-parallel \
     --with-zlib=/projects/albany/intel5.1 --prefix=/projects/albany/intel5.1
  
    make -j 4
    make install
  

Building parallel-netcdf

• Version 1.8.1

• Configure and build:

    ./configure --prefix=/projects/albany/intel5.1 CC=mpiicc F77=mpiifort F90=mpiifort CXX=mpiicpc \
    CXXFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
    CFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
    FFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
    FCFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
    --with-mpi=/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64
  
    make -j 8 
    make install
  

Building Netcdf

• Version 4.4.1.1

• Edit the file include/netcdf.h. Make the following edits near line 265:

    #define NC_MAX_DIMS    65536    /* max dimensions per file */
    #define NC_MAX_ATTRS    8192
    #define NC_MAX_VARS   524288    /* max variables per file */
    #define NC_MAX_NAME   256
    #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
  

• Add the install directory of the hdf5 shared libs and the libpnetcdf.so file to your LD_LIBRARY_PATH or the config below will fail:

    export LD_LIBRARY_PATH=/projects/albany/intel5.1/lib:$LD_LIBRARY_PATH	
  

• Now, configure using CMake and build:

    ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc \
        CXXFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
        CFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
        LDFLAGS="-fPIC -L/projects/albany/intel5.1/lib -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
        FCFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
        --prefix=/projects/albany/intel5.1 --enable-netcdf-4 --enable-pnetcdf
  
    make -j 8
    make install
  

Building boost

• Version 1.63.0

• Build boost with the commands

    echo 'using intel-linux : : /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/bin/intel64/icpc : <cxxflags>"-O3 -DNDEBUG -diag-disable=cpu-dispatch" ;' > ~/user-config.jam
    echo 'using mpi : /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64/bin/mpiicpc : <cxxflags>"-I/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64/include -O3 -DNDEBUG -diag-disable=cpu-dispatch" ;' >> ~/user-config.jam
    ./bootstrap.sh --with-toolset=intel-linux --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany/intel5.1
    ./b2 toolset=intel-linux -j 4
    ./b2 toolset=intel-linux -j 4 install
  

• The above bjam command gave me an error, so I removed the following lines from project-config.jam:

    if ! intel-linux in [ feature.values <toolset> ]
    {
        using intel-linux ;
    }
  

Building ParMetis

• Version 4.0.3

    wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
    tar -xvf parmetis-4.0.3.tar.gz
  

• Edit metis.h to use an IDXTYPEWIDTH of 64

    cd parmetis-4.0.3/metis/include
    vi metis.h
    #define IDXTYPEWIDTH 64
    #define REALTYPEWIDTH 64
  

• Run CMake to build parmetis

    cd /projects/albany/src/parmetis-4.0.3 
    cd build
    export metis=/projects/albany/src/parmetis-4.0.3/metis/  
    cmake \
    -DCMAKE_INSTALL_PREFIX=/projects/albany/intel5.1 \
    -DMETIS_PATH=$metis \
    -DGKLIB_PATH=$metis/GKlib \
    -DCMAKE_C_COMPILER=mpiicc \
    -DCMAKE_CXX_COMPILER=mpiicpc \
    -DCMAKE_C_FLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -fPIC" \
    -DCMAKE_CXX_FLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -fPIC" \
    ..
    make -j 4
    make install
  
    cp libmetis/libmetis.a /projects/albany/intel5.1/lib
    cp $metis/include/metis.h /projects/albany/intel5.1/include
  

SuperLU (optional)

• Version 4.3

• Build paths

    mkdir /projects/albany/intel5.1/SuperLU_4.3
    mkdir /projects/albany/intel5.1/SuperLU_4.3/include
    mkdir /projects/albany/intel5.1/SuperLU_4.3/lib
  

• Edit make.inc for your machine and environment

    PLAT = _linux
    SuperLUroot = /projects/albany/intel5.1/SuperLU_4.3
    SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
    TMGLIB          = libtmglib.a
    MKLHOME         = /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64
    BLASLIB     = -L${MKLHOME} -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
    LIBS        = $(SUPERLULIB) $(BLASLIB)
    ARCH         = xiar
    ARCHFLAGS    = cr
    RANLIB       = ranlib
    CC           = mpiicc 
    CFLAGS       = -O3 -DNDEBUG -DPRNTlevel=0 -DAdd_ -mkl=sequential
    NOOPTS       =
    FORTRAN      = mpiifort
    FFLAGS       = -O3 -mkl=sequential
    LOADER       = $(CC)
    LOADOPTS     =
    MATLAB       = /usr/sww/matlab
  

• Build SuperLU

    make
    cd SRC
    cp *.h /projects/albany/intel5.1/SuperLU_4.3/include		
  

• Note: Do not make SuperLU in parallel (-j * make argument) as the build will be incomplete.

Build HWLOC

• HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

• Version 1.11.6

    ./configure CC=mpiicc CXX=mpiicpc --prefix=/projects/albany/intel5.1
    make -j 4
    make install