Modules Issue for HAFS on Orion

[AndrewHazelton]

Hello,

It seems like there is a conflict between the modulefiles on Orion and the forecast executable. Both Mu-Chieh Ko and I have gotten an error of the form:

MPIDI_OFI_mpi_init_hook(1382): OFI get address vector map failed)
[1665683994.060988] [Orion-25-40:224966:0] select.c:438 UCX ERROR no active messages transport to : posix/memory - Destination is unreachable, sysv/memory - Destination is unreachable, self/memory - Destination is unreachable, sockcm/sockaddr - no am bcopy, dc_mlx5/mlx5_0:1 - Destination is unreachable, rdmacm/sockaddr - no am bcopy, cma/memory - no
Abort(1091471) on node 1243 (rank 1243 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:

With some tweaks to the module file, this error goes away, but the following error shows up:

Lmod has detected the following error: These module(s) or extension(s) exist

but cannot be loaded as requested: "w3nco/2.4.1"

Try: "module spider w3nco/2.4.1" to see how to load the module(s).

The tests I am running are working with this branch: * feature/hafsv0.3_ensda_eps

Thanks!

Andy Hazelton

[ghassan-alaka]

This is the code block that allowed the fgat tasks to succeed on Orion. This issue appears to affect some of the newer builds. However, I believe Lew built HAFS around 9/20 and he is not experiencing these module issues. So, this is a temporary solution until the modules at all levels of the HAFS workflow have been unified. I am not planning to commit this change.

if [ "${RUN_ENVIR^^}" != NCO ]; then
module list
#module purge
#module unload intelpython
module use ${HOMEhafs}/sorc/hafs_forecast.fd/modulefiles
if [ "${machine}" = "wcoss_cray" ] || [ "${machine}" = "wcoss_dell_p3" ]; then
module load ufs_${machine}
else
module load ufs_${machine}.intel
fi
module list
export NDATE="/apps/contrib/NCEP/libs/hpc-stack/intel-2018.4/prod_util/1.2.2/bin/ndate"
fi

[AndrewHazelton]

Thanks for this suggestion! This seems to take care of most of the issues, although I still see occasional UCM/MPI errors pop up in the forecast task. A rewind seems to always take care of them, but something to note.

[panll]

I also see the similar error when I ran the develop branch on Orion: MPIDI_OFI_mpi_init_hook(1382): OFI get address vector map failed)
When reran it, and it passed without error. This problem seems relate to MPI and UCX setting. I'll send an email to Orion help desk to see whether they could have some comments and suggestions. Thanks!

[panll]

The Orion help desk suggests that errors come from inconsistency in module use.