Skip to content

Rhel6 using Gcc

Glen Hansen edited this page Feb 11, 2019 · 6 revisions

This set of build instructions is for the Albany suite on Rhel6 linux.

The compilers are gcc 5.2.0, OpenMPI 1.8.5, and RHEL 6 linux.

Modules on CEE:

	module load sierra-devel/gcc-5.2.0-openmpi-1.8.8
	module load sierra-mkl/16.0-2016.2.181

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

  • The TPLs are installed in /projects/albany

Building ZLib

  • Version 1.2.8

      CC=mpicc CFLAGS=-O3 ./configure --64 --archs="-arch x86_64" --prefix=/projects/albany
      make -j 4
      make install
    

Building HDF5

  • Version 1.8.17

      ./configure CC=mpicc FC=mpifort CXX=mpicxx CXXFLAGS="-O3 -fPIC -march=native" \
       CFLAGS="-O3 -fPIC -march=native" \
       FCFLAGS="-O3 -fPIC -march=native" --enable-parallel \
       --with-zlib=/projects/albany --prefix=/projects/albany
      make -j 4
      make install
    

Building parallel-netcdf

  • Version 1.7.0

  • Configure and build:

      ./configure --prefix=/projects/albany MPICC=mpicc MPIF77=mpifort MPIF90=mpifort MPICXX=mpicxx \
      CXXFLAGS="-fPIC -O3 -march=native" CFLAGS="-fPIC -O3 -march=native" \
      F77FLAGS="-fPIC -O3 -march=native" F90FLAGS="-fPIC -O3 -march=native"
      make -j 8 
      make install
    

Building Netcdf

  • Version 4.4.0

  • Edit the file include/netcdf.h. Make the following edits near line 265:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
      #define NC_MAX_ATTRS    8192
      #define NC_MAX_VARS   524288    /* max variables per file */
      #define NC_MAX_NAME   256
      #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
    
  • Now, configure using CMake and build:

      ./configure CC=mpicc FC=mpifort CXX=mpicxx \
          CXXFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
          CFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
          LDFLAGS="-fPIC -L/projects/albany/lib -O3 -march=native" \
          FCFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
          --prefix=/projects/albany --disable-doxygen --enable-netcdf4 --enable-pnetcdf
      make -j 8
      make install
    

Building boost

  • Version 1.55.0

  • Build boost with the commands

      echo "using mpi : /sierra/sntools/SDK/mpi/openmpi/1.8.5-gcc-5.2.0-RHEL6/bin/mpicxx ;" >> ./tools/build/v2/user-config.jam
      echo "using gcc : /sierra/sntools/SDK/compilers/gcc/5.2.0-RHEL6/bin/g++ ;" >> ./tools/build/v2/user-config.jam
      ./bootstrap.sh --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany
      ./b2 -j 4
      ./b2 -j 4 install
    

Building ParMetis

  • Version 4.0.3

      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
      tar -xvf parmetis-4.0.3.tar.gz
    
  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd parmetis-4.0.3/metis/include
      vi metis.h
      #define IDXTYPEWIDTH 64
      #define REALTYPEWIDTH 64
    
  • Run CMake to build parmetis

      cd build
      export metis=/projects/albany/src/parmetis-4.0.3/metis/  
      cmake \
      -DCMAKE_INSTALL_PREFIX=/projects/albany/parmetis-4.0.3 \
      -DMETIS_PATH=$metis \
      -DGKLIB_PATH=$metis/GKlib \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_CXX_COMPILER=mpicxx \
      -DCMAKE_C_FLAGS=-O3  \
      -DCMAKE_CXX_FLAGS=-O3  \
      ..
      make -j 4
      make install
    
      cp libmetis/libmetis.a /projects/albany/parmetis-4.0.3/lib
      cp $metis/include/metis.h /projects/albany/parmetis-4.0.3/include
    

SuperLU (optional)

  • Version 4.3

  • Build paths

      mkdir /projects/albany/SuperLU_4.3
      mkdir /projects/albany/SuperLU_4.3/include
      mkdir /projects/albany/SuperLU_4.3/lib
    
  • Edit make.inc for your machine and environment

      PLAT = _linux
      SuperLUroot = /projects/albany/SuperLU_4.3
      SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
      TMGLIB          = libtmglib.a
      BLASDEF     = -DUSE_VENDOR_BLAS
      BLASLIB     = -L/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
      LIBS        = $(SUPERLULIB) $(BLASLIB)
      ARCH         = ar
      ARCHFLAGS    = cr
      RANLIB       = ranlib
      CC           = gcc
      CFLAGS       = -O3
      NOOPTS       =
      FORTRAN      = gfortran
      FFLAGS       = -O3
      LOADER       = $(CC)
      LOADOPTS     =
      CDEFS        = -DAdd_
      MATLAB       = /usr/sww/matlab
    
  • Build SuperLU

      make -j 4
      cd SRC
      cp *.h /projects/albany/SuperLU_4.3/include		
    

Build HWLOC

  • HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

  • Version 1.8

      ./configure CC=mpicc CXX=mpicxx --prefix=/projects/albany
      make -j 4
      make install
    

Build Trilinos

Notes: In this example

  • The Trilinos source is installed in /home/gahanse/Codes/Trilinos, using a command like

      cd /home/gahanse/Codes
      git clone [email protected]:trilinos/Trilinos.git
    
  • The SCOREC tools are installed as a subdirectory of Trilinos, and are build automatically by the Trilinos build process. The commands to do this are

      cd /home/gahanse/Codes/Trilinos
      git clone https://github.com/SCOREC/core.git SCOREC
    
  • do-configure script

      #!/bin/sh
      #
      export TRILINOS_HOME=/projects/albany/src/Trilinos
      BUILD_DIR=`pwd`
      INSTALL_DIR=/projects/albany/trilinos/MPI_REL
      BOOST_DIR=/projects/albany
      LIB_DIR=/projects/albany
      MPI_BASE_DIR=/sierra/sntools/SDK/mpi/openmpi/1.8.8-gcc-5.2.0-RHEL6
      NETCDF=/projects/albany
      HDFDIR=/projects/albany
      PARMETISDIR=/projects/albany
      MKL_PATH=/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210
      HWLOC_PATH=/projects/albany
      LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/libmkl_core.a $MKL_PATH/mkl/lib/intel64/libmkl_sequential.a -Wl,--end-group"
      
      cmake \
          -Wno-dev \
          -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
          -D Trilinos_ENABLE_SCOREC:BOOL=ON \
          -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
          -D CMAKE_BUILD_TYPE:STRING=NONE \
          -D CMAKE_CXX_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
          -D CMAKE_C_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
          -D CMAKE_Fortran_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
          -D Rythmos_ENABLE_DEBUG:BOOL=ON \
          -D KokkosClassic_DefaultNode:STRING="Kokkos::Compat::KokkosSerialWrapperNode" \
          -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
          -D Tpetra_INST_INT_LONG_LONG:BOOL=ON \
          -D Tpetra_INST_INT_INT:BOOL=ON \
          -D Tpetra_INST_DOUBLE:BOOL=ON \
          -D Tpetra_INST_FLOAT:BOOL=OFF \
          -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \
          -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \
          -D Tpetra_INST_INT_LONG:BOOL=OFF \
          -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \
          -D Trilinos_ENABLE_Kokkos:BOOL=ON \
          -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
          -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \
          -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
          -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
          -D Kokkos_ENABLE_Serial:BOOL=ON \
          -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
          -D Kokkos_ENABLE_Pthread:BOOL=OFF \
      \
          -D TPL_ENABLE_MPI:BOOL=ON \
          -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
      \
      	-D TPL_ENABLE_Pthread:BOOL=OFF \
      \
          -D TPL_ENABLE_Boost:BOOL=ON \
          -D TPL_ENABLE_BoostLib:BOOL=ON \
          -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
      	-D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      	-D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      	-D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      	-D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      	-D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      	-D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      \
          -D TPL_ENABLE_Netcdf:STRING=ON \
      	-D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
      	-D Netcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
      \
          -D TPL_ENABLE_HDF5:STRING=ON \
          -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
          -D HDF5_LIBRARY_DIRS:PATH=$HDFDIR/lib \
      \
          -D TPL_ENABLE_Zlib:STRING=ON \
          -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \
          -D Zlib_LIBRARY_DIRS:PATH=$HDFDIR/lib \
      \
          -D TPL_ENABLE_BLAS:STRING=ON \
          -D TPL_ENABLE_LAPACK:STRING=ON \
          -D TPL_BLAS_LIBRARIES:STRING="$LABLAS_LIBRARIES" \
          -D TPL_LAPACK_LIBRARIES:STRING="$LABLAS_LIBRARIES" \
      \
          -D TPL_ENABLE_ParMETIS:STRING=ON \
          -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
          -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
      \
      	-D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
          -D Trilinos_EXTRA_LINK_FLAGS:STRING="-L${HDFDIR}/lib -lnetcdf -lhdf5_hl -lhdf5 -lz -lgfortran -Wl,-rpath,/projects/albany/lib" \
      \
          -D Trilinos_ENABLE_Moertel:BOOL=ON \
          -D Moertel_ENABLE_EXAMPLES:BOOL=ON \
          -D Moertel_ENABLE_Morkon:BOOL=ON \
          -D Trilinos_ENABLE_TriKota:BOOL=OFF \
          -D HAVE_INTREPID_KOKKOSCORE:BOOL=ON \
          -D Trilinos_ENABLE_SECONDARY_TESTED_CODE=OFF \
          -D Trilinos_ENABLE_Amesos2:BOOL=ON \
          -D Amesos2_ENABLE_KLU2:BOOL=ON \
      \
      $EXTRA_ARGS \
      ${TRILINOS_HOME}
    

To build Trilinos, use an out-of-source build. I usually make a build directory that matches the configuration of Trilinos that I am building in the Trilinos root directory:

	cd /home/gahanse/Codes/Trilinos
    mkdir build
	cd build
	mkdir MPI_REL
  • Place the above do-configure file into the MPI_REL directory, change directories into MPI_REL, and perform the configure, build, and install:

      cd MPI_REL
      ./do-configure
      make -j 4
      make -j 4 install
    

Build Albany

  • CMake configure script:

      #!/bin/sh
      
      export ALB_64BIT_INT=OFF
      export ALB_EPETRA=ON
      export ALB_ENABLE_SCOREC=ON
      export ALB_ENABLE_LCM=ON
      export ALB_ENABLE_LAME=OFF
      export ALB_ENABLE_HYDRIDE=OFF
      export ALB_ENABLE_QCAD=OFF
      export ALB_ENABLE_MOR=OFF
      export ALB_ENABLE_ASCR=OFF
      export ALB_ENABLE_AERAS=ON
      export ALB_ENABLE_FELIX=ON
      export ALB_ENABLE_AlbanyCI=OFF
      export ALB_DEBUG=OFF
      export ALB_ENABLE_CHECK_FPE=OFF
      export ALB_ENABLE_SPECULATIVE=OFF
      export ALB_ENABLE_SG_MP=OFF
      export ALB_ENABLE_PERF_TESTS=ON
      export TRILINOS_INSTALL_DIR=/projects/albany/trilinos/MPI_REL
      export ALB_VERBOSE=OFF
      
      cmake \
            -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
            -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
            -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
            -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
            -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
            -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
            -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
            -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
            -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
            -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
            -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
            -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
            -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
            -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
            -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
            -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
            -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
            -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
            -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \
      \
      ..
    

Clone the Albany repo in a suitable place:

	cd /home/ghansen/Codes
	git clone https://github.com/SNLComputation/Albany.git
  • Create an out-of-source build directory

      cd Albany
      mkdir build
    
  • Place the above configure script into the build directory, then configure, build, and test

      cd build
      ./buildAlbany
      make -j 4
      ctest
    
Clone this wiki locally