Clarification of use of various (external) libraries for cesm-coupling

[ShervanGharari]

Below is based on my non-expert understanding, apologize in advance if not accurate.
Currently, the Makefile for cesm-coupling branch assumes the structure libraries that are prepared by ./manage_externals/checkout_externals. This perhaps results in mizuRoute Makefile, or the Makefile for libraries, assuming a relative path to each of the needed files during compilation. Tracking each of the libraries also gets very complex across many existing Makefiles.
A way forward can be to specify all the packages, similar to netcdf for example, in mizuRoute Makefile. This link can be for gptl or parallelio. The Makefile can be directly pointed at the location of the libraries if they are built prior to the compilation of mizuRoute. This also makes it perhaps easier to link the Makefile to libraries prepared by ./manage_externals/checkout_externals.

[ShervanGharari]

Perhaps a bit of more explanation of what I am after; I would like to see the path from pre-built libraries in the mizuRoute Makefile, sth like:

PNETCDF = path/to/parallel/netcdf # which includes lib/libpnetcdf.a
NETCDFC = path/to/netcdf/c # which includes lib/libnetcdf.a
NETCDFF = path/to/netcdf/fortran # which includes lib/libnetcdff.a
GPTL = path/to/gptl # which includes lib/libgptl.a and lib/libgptlf.a
PIO = path/to/parallel/io # which includes lib/libpioc.a and lib/libpiof.a

in case the user sets the flags to true or yes for PIO or GPTL and the paths are not provided, makefile can be directed to externals. If the paths are provided, it is expected that mizuRoute src is built against the provided pre-built libraries.
Would that be a feasible option?

[nmizukami]

Hi Erik (@ekluzek), any comments on this?? maybe we will need to talk to Brian (Dobbin)??

[ShervanGharari]

More information on what I am currently trying to achieve. I am trying to create a dockerfile with a spack environment inside it. I managed to clone and build the mizuRoute main branch with some minor modifications to the makefile, however, I am not successful with the cesm-coupling branch. I highly suspect this is because of the interconnected path for the libraries that are populated by checkout_extrenals and need to be built during the compilation of mizuRoute. The dockerfile looks like this:

# Use Ubuntu as the base image
FROM ubuntu:20.04 as builder

# Set environment variables
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y \
    build-essential \
    ca-certificates \
    coreutils \
    curl \
    environment-modules \
    gfortran \
    gpg \
    lsb-release \
    python3 \
    python3-distutils \
    python3-pip \
    python3-venv \
    unzip \
    zip \
    git \
    nano \
    wget

# Set up Spack environment
RUN rm -rf /opt/spack-environment
RUN mkdir -p /opt/spack-environment
RUN (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - zlib" \
&&   echo "  - openmpi" \
&&   echo "  - netcdf-c" \
&&   echo "  - netcdf-fortran" \
&&   echo "  - parallel-netcdf" \
&&   echo "  - parallelio" \
&&   echo "  - gptl" \
&&   echo "  concretizer:" \
&&   echo "    unify: true" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software" \
&&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml

# Install Spack and the specs
RUN rm -rf /opt/spack && \
    git clone -c feature.manyFiles=true https://github.com/spack/spack.git /opt/spack && \
    . /opt/spack/share/spack/setup-env.sh && \
    cd /opt/spack-environment && \
    spack env activate . && spack install --fail-fast && spack gc -y

# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s

# Set up environment modifications
RUN . /opt/spack/share/spack/setup-env.sh && \
    spack env activate --sh -d /opt/spack-environment >> /etc/profile.d/z10_spack_environment.sh

# Bare OS image to run the installed executables
FROM ubuntu:20.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /opt/spack /opt/spack
COPY --from=builder /usr /usr
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh

# Activate Spack environment and load packages
. /etc/profile.d/z10_spack_environment.sh && \
. /opt/spack/share/spack/setup-env.sh && \
spack env activate /opt/spack-environment -p

ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]

after building the image and run the container:

docker build -t spacktest .
docker run -it spacktest

or restart existing container after running the target container in docker desktop

docker exec -it <hash> bash

after running the docker container, activating the spack env

. /etc/profile.d/z10_spack_environment.sh
. /opt/spack/share/spack/setup-env.sh
spack env activate /opt/spack-environment -p

and try to see where the packages are installed

spack find -p netcdf-fortran

inside the library can be check by cd to the path of spack build location

cd $(spack location -i netcdf-c)/lib

I believe if the path are separated in the makefile I should be able to first manually link each of the built packages to the makefile and hopefully compile mizuRoute inside the container (similar to the step I took for compiling the main branch).

[nmizukami]

Ok, I tested compiling with system installed parallelio, and get it compiled.

The timing functions used in mizuRoute does not directly use the GPTL timing functions, but use the module built in parallelio that wrap GPTL functions (the module called perf_mod, and functions like t_prf, t_finalizedf). So If I try to use the system installed gptl, the syntax is not matched and is not be compiled. So gptl library needs to be built with parallelio, then mizuRoute should be compiled.

For me, I cannot re-build parallelio with gptl on NCAR HPC (or maybe I could ask if it exists), so I commented out all the timing functions from perf_mod, and excluded gptl, then compiled.

So I changed the makefile this portion (and removed yes from isGPTL =)

223 #========================================================================
224 # External libaries that might need to be built
225 #========================================================================
226 #LIBDIR       = $(F_MASTER)build/lib
227 #PIOLIBDIR    = $(LIBDIR)/piolib/lib
228 #PIOINCDIR    = $(LIBDIR)/piolib/include
229 #PIOLIB       = $(PIOLIBDIR)/libpiof.a $(PIOLIBDIR)/libpioc.a
230 #MPISERLIBDIR = $(LIBDIR)/mpi-seriallib
231 #MPISERLIB    = $(MPISERLIBDIR)/libmpi-serial.a
232 
233 PIOLIBDIR= $(NCAR_ROOT_PARALLELIO)
234 PIOINCDIR= $(NCAR_ROOT_PARALLELIO)/include
235 PIOLIB   = $(PIOLIBDIR)/lib/libpiof.so $(PIOLIBDIR)/lib/libpioc.so
236 GPTLLIBDIR=$(NCAR_ROOT_GPTL)
237 GPTLINCDIR=$(NCAR_ROOT_GPTL)/include
238 GPTLLIB   =$(GPTLLIBDIR)/lib/libgptl.so
239 
240 
241 ifeq "$(isPIO)" "yes"
242   EXTINCLUDES += -I$(PIOINCDIR)
243   EXTLIBS += $(PIOLIB)
244   LDFLAGS += -L$(PIOLIBDIR)/lib -lpiof -lpioc
245 
246   ifeq "$(isGPTL)" "yes"
247 #     LDFLAGS += -lgptl
248      EXTINCLUDES += -I$(GPTLINCDIR)
249      EXTLIBS += $(GPTLLIB)
250      LDFLAGS += -L$(GPTLLIBDIR)/lib -lgptl
251   endif
252 endif

I am not sure what the best way to handle gptl so it works for both stand-alone and cesm (cesm use gptl from parallelio)

[ekluzek]

#461 removed uneeded externals (mpi-serial, mct, cime), so it clarifies a lot of this. So only PIO is needed now.

The GPTL library is still an issue though because the PIO build needs to have it built with GPTL enabled in order to use it. So building it within the mizuRoute build always works, but using a pre-built library requires that library to have GPTL enabled, and that isn't always the case.