Unit test fms2io fails with ifx

[laurenchilutti]

Describe the bug
The fms2io unit test test_fms2_io.sh 2 is failing with the intel IFX compiler. This seems to be related to mpp_gather calls in fms2io.

The FMS commit being referred to by the backtrace here is f0ec93f
The backtrace is:
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread-2.28.s 00007F9AED16ECF0 Unknown Unknown Unknown
test_compressed_w 0000000000429D3D Unknown Unknown Unknown
libFMS.so.16.0.0 00007F9AF276734F compressed_write_ 407 compressed_write.inc
libFMS.so.16.0.0 00007F9AF2775D5A compressed_write_ 714 compressed_write.inc
test_compressed_w 0000000000413DC8 write_data_wrappe 157 test_compressed_writes.F90
test_compressed_w 000000000040EA83 test_compressed_w 79 test_compressed_writes.F90
test_compressed_w 000000000040DAFD Unknown Unknown Unknown
libc-2.28.so 00007F9AEC98BD85 __libc_start_main Unknown Unknown
test_compressed_w 000000000040DA1E Unknown Unknown Unknown

Line 407 is a call to mpp_gather.

I have verified that all of the inputs to the mpp_gather function are allocated and have valid values.

To Reproduce
On GFDL AMD Dev Box:

module load oneapi/2023.1 compiler mpi hdf5 netcdf libyaml

export CC="mpiicc -cc=icx"
export FC="mpiifort -fc=ifx"
export CFLAGS="`nc-config --cflags` -O0 -g -traceback"
export FCFLAGS="`nc-config --cflags` -O0 -g -traceback -check all"

make
cd test_fms
make check -k

you will see:
FAIL: test_fms2_io.sh 2 - Compressed writes tests

To get a backtrace for this test I did:
export LD_LIBRARY_PATH="<path-to-FMS-builddir>/libFMS/.libs/:${LD_LIBRARY_PATH}"
cd test_fms/fms2_io/test_fms2_io-files
mpirun -n 5 ../.libs/test_compressed_writes

Expected behavior
No tests fail

System Environment
On GFDL AMD Dev Box:

module load oneapi/2023.1 compiler mpi hdf5 netcdf libyaml

export CC="mpiicc -cc=icx"
export FC="mpiifort -fc=ifx"
export CFLAGS="`nc-config --cflags` -O0 -g -traceback"
export FCFLAGS="`nc-config --cflags` -O0 -g -traceback -check all"

Additional context
This was discovered in the FMS compiler project. As that project is closing we are tracking this with an issue in FMS and will still work to narrow down the root cause of the test failure.