Nightly Linux builds are not very backwards compatible

[mwtoews]

Hi, checking to see if anyone else has experienced issues using modflow6-nightly-build for Linux using a recent (neither latest or old) Ubuntu 20.04. This can be replicated using docker:

docker run -it --rm ubuntu:20.04 bash

with these commands:

apt-get update -q && apt-get install -y -q python3 wget
wget -q https://raw.githubusercontent.com/modflowpy/flopy/develop/flopy/utils/get_modflow.py
python3 get_modflow.py --repo modflow6-nightly-build :system
mf6

which raises:

mf6: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by mf6)

similar with ldd $(which mf6)

/usr/local/bin/mf6: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/local/bin/mf6)
	linux-vdso.so.1 (0x00007ffe1b1dc000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5effdb8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5efffae000)

There are no issues using the latest Ubuntu 22.04 (docker image ubuntu:22.04), which has GLIBC 2.35. And there are no issues with the default executables (releases).

I'm not sure how modflow6-nightly-build works, but it should be modified to compile Linux executables that are more backwards compatible. There are many projects out there that aim to provide this, e.g. https://phusion.github.io/holy-build-box/ (used for PESTPP), or https://github.com/pypa/manylinux (used for Python binary wheels distributed on PyPI).

[wpbonelli]

Thanks for raising this @mwtoews, we have seen similar glibc issues. Last this was discussed, IIRC, the thought was to keep the executables distro as broadly/backwards compatible as possible (hence building on Ubuntu 20.04) while the bleeding edge release would build on the latest available version of each platform, caveats included. Certainly open to suggestions on this approach. And if the build tools you linked would work as a drop-in solution for better compatibility, that seems great.

[wpbonelli]

Took a brief look at HBB. It is a Docker image which allows defining a tightly controlled build environment independently of the build host. GitHub allows jobs to run in a user-specified Docker image, so the HBB image could be subbed for Ubuntu 22.04. However the HBB image would need to be modified to install Meson, compilers, etc.

I noticed HBB mentions Meson in the README. There seems to be overlap in scope. Their respective developers have had some interaction. Meson's recommendation to avoid glibc incompatibility is to build on a system at least as old as the systems to be supported. The easiest option for wider Linux compatibility may just be building on an earlier Ubuntu (unless e.g. CentOS provides broader support)

If this is done, is it worthwhile to keep a canary build on the latest Ubuntu release? May help avoid surprises when it comes time to update

[mwtoews]

Some ideas are to create a new Docker image based on phusion/holy-build-box-64:latest that has the necessary tools installed, which can be used to build mf6. One big question is if it's legally possible to build and save a docker image with Intel Fortran. To get you going with HBB, start an interactive Docker session using docker run -t -i --rm phusion/holy-build-box-64:latest bash then run:

source /hbb_exe/activate
yum install meson

(except that CentOS 7's version of meson is too old for MF6, so other workarounds are needed to get it ready, like python3 -m pip install meson.) If you're interested, here is pestpp's build script using CMake (via usgs/pestpp#62).

[mwtoews]

Found a partial workaround from Ubuntu 20.04.

Step one. Download and install newer glibc-2.34:

cd /tmp
wget http://ftp.gnu.org/gnu/libc/glibc-2.34.tar.bz2
tar xfj glibc-2.34.tar.bz2
mkdir glibc-2.34-build
cd glibc-2.34-build
../glibc-2.34/configure --prefix=/opt/glibc-2.34
make -j
sudo make install

Step two. Install PatchELF to modify the binary:

sudo apt install patchelf

Step three. Modify the mf6 binary:

patchelf --set-interpreter /opt/glibc-2.34/lib/ld-linux-x86-64.so.2 --set-rpath /opt/glibc-2.34/lib:/usr/lib64 $(which mf6)

Unfortunately, libmf6.so cannot easily be patched to be used with ctypes (xmipy, modflowapi).

patchelf --set-rpath /opt/glibc-2.34/lib:/usr/lib64 /path/to/libmf6.so
python -c "from ctypes import CDLL; CDLL('/path/to/libmf6.so')"
# OSError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /path/to/libmf6.so)

and this is related to RPATH priorities and how Python was compiled (ref).

[wpbonelli]

with 753fdf7 the nightly linux build should be more broadly compatible (last night's tested on ubuntu 18.04)