Jobs failing due to "unreachable" EPIC modules

[jak5808]

System: Orion
HAFS version: this issue has occurred across different HAFS branches, including develop

Last month, a number of module libraries were moved from /work/noaa/epic-ps to a new location (/work/noaa/epic/) after the epic-ps directory that contained old libraries was removed. Ever since that move happened, many of my jobs will fail due to the modules being "unreachable". I can rewind the jobs until they succeed, but it can take as many as 4 or 5 rewinds before the modules behave, which is a pain for ensemble jobs. The modules fail as soon as the executable is called with the following error messages included:

[Orion-19-42:369850:0:369850] address.c:1052 Assertion `*addr_version == UCP_OBJECT_VERSION_V2' failed: addr version 9

[1700529122.272534] [Orion-25-40:65147:0] select.c:438 UCX ERROR no active messages transport to : self/memory - Destination is unreachable, dc_mlx5/mlx5_0:1 - Destination is unreachable, rdmacm/sockaddr - no am bcopy

Error in system call pthread_mutex_destroy: Device or resource busy
../../src/mpi/init/init_thread_cs.c:66
Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(143)........:
MPID_Init(1310)..............:
MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed
In: PMI_Abort(1090703, Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(143)........:
MPID_Init(1310)..............:
MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed)

An example log file can be found here:
/work2/noaa/aoml-hafsda/knisely/HAFS_May_23_hfsb_dualres_ensda_olbc_augens06/2022082700/00L/hafs_forecast_ens010.log.0

I am currently running these experiments again with a clean .bashrc to see if the issue is due to conflicting modules/environmental settings, however I have talked to multiple fellow community members who are experiencing this too, so I think it's unlikely this is the issue. Please let me know if you would like any more information from me.

Thanks,
Joey

[natalie-perlin]

Hi Joey,

Thank you for reaching out. I'm sure it could be solved quickly. What is the path of the software stack on Orion modules that you use?
Could you please open a permission to view files in your /work2/noaa/aoml-hafsda/knisely/HAFS_May_23_hfsb_dualres_ensda_olbc_augens06/2022082700/00L directory?

[jak5808]

Hi Natalie,

The path of the software stack is /work/noaa/epic/role-epic/contrib/orion/hpc-stack/intel-2022.1.2/modulefiles/stack/

I also opened permissions to this directory.

Thanks,
Joey

[natalie-perlin]

Please make sure your HAFS modulefile has the following location of the software stack on Orion:
/work/noaa/epic/role-epic/contrib/orion/hpc-stack/intel-2022.1.2/modulefiles/stack

[natalie-perlin]

Hi Joey,
Thank you for the response. I looked into your log file. The issue looks more with the MPI initialization in general. Could you print out the list of modules loaded when building the executable hafs_forecast.x, and the list of modules loaded during the runtime?

[jak5808]

Here is the list of modules loaded during forecast build, as found in /work/noaa/aoml-hafsda/knisely/HAFS_Nov_23/sorc/logs/build_forecast.log

Currently Loaded Modules:

1. contrib/0.1 5) hpc/1.2.0 9) hpc-impi/2022.1.2 13) hdf5/1.10.6 17) fms/2022.04 21) g2tmpl/1.10.2 25) gftl-shared/v1.5.0
2. noaatools/3.1 6) intel/2022.1.2 10) jasper/2.0.25 14) netcdf/4.7.4 18) bacio/2.4.1 22) ip/3.3.3 26) mapl/2.22.0-esmf-8.3.0b09
3. cmake/3.22.1 7) hpc-intel/2022.1.2 11) zlib/1.2.11 15) pio/2.5.7 19) crtm/2.4.0 23) sp/2.3.3 27) ufs_common
4. python/3.7.5 8) impi/2022.1.2 12) libpng/1.6.37 16) esmf/8.3.0b09 20) g2/3.4.5 24) w3emc/2.9.2 28) ufs_orion.intel

I'm not sure what you mean by "modules loaded during the runtime," can you elaborate?

[natalie-perlin]

Joey,
Thanks for the list. How is your job being submitted? Is it a bash script? Does it load all the modules before calling an executable?
Could you place a "module list" statement somewhere immediately before the model executable is called?

[jak5808]

Natalie,

Yes, I use the following bash script (as an example): /work/noaa/aoml-hafsda/knisely/HAFS_Nov_23/rocoto/hafs_test1.sh

This script does "source /work/noaa/aoml-hafsda/knisely/HAFS_Nov_23/ush/hafs_pre_job.sh.inc" to load the necessary modules, including modules from /work/noaa/epic/role-epic/contrib/orion/hpc-stack/intel-2022.1.2/modulefiles/stack

Jobs are then submitted through the workflow management system rocoto with the command "rocotorun"

Let me know if you need more information from me, or if I misunderstood something. Thanks!

[gspetro-NOAA]

Hi @jak5808 ,

I checked in with our Software Integration (SI) Team, who has been working with you, and they are working on troubleshooting this issue, but they may not be able to get back to you until after the holidays since many team members are off this week and/or next. I'm hopeful they will get you an answer sooner rather than later, but in case they don't/can't, I wanted to make sure you know that we haven't forgotten your question!

Best,
Gillian Petro

[natalie-perlin]

@jak5808 -
Joey,
My apologies for delay in responding to your questions on HAFS runtime error.
Please see my notes after looking into your scripts on building and running HAFS.
The building of the HAFS code seems to be done correctly. However, the runtime logfile shows some things of concern.
The runscript /work/noaa/aoml-hafsda/knisely/HAFS_Nov_23/ush/hafs_test1.sh
needs to start MPI jobs, and supposedly used another input file containing SLURM directives, such as srun to start a multi-process and multi-tasking (MPI) executable. The log file (your example of /work2/noaa/aoml-hafsda/knisely/HAFS_May_23_hfsb_dualres_ensda_olbc_augens06/2022082700/00L/hafs_forecast_ens010.log.0 ) shows the following call after which the forecast seem to die:

srun --mem=0 --ntasks=1080 --ntasks-per-node=20 --cpus-per-task=2 ./hafs_forecast.x

Is it the correct domain decomposition?
Where the domain decomposition is being set? Do you have any newer attempts to run the executable built in Nov. 2023? The runtime log file uses directory .../HAFS_May_23/... , while the newer executable is built under .../HAFS_Nov_23/... directory structure. Could this be a confusion of names ("May" vs "Nov")?..

[gspetro-NOAA]

Hi @jak5808 ,

Just wanted to check in and see if this issue got resolved or if you have further questions for @natalie-perlin ?

Thanks,
Gillian

[jak5808]

Hey Gillian, Thanks for checking in. I missed Natalie's last response, for which I apologize, but at this point it seems like the issue has been resolved. I have not received any similar error messages recently, although I will keep an eye out. Did the SI Team figure out what was going wrong? I haven't changed any of my model configurations or domain decompositions so it was nothing on my end. Thanks again, Joey

…

On Mon, Feb 12, 2024 at 11:03 AM Gillian Petro ***@***.***> wrote: Hi @jak5808 <https://github.com/jak5808> , Just wanted to check in and see if this issue got resolved or if you have further questions for @natalie-perlin <https://github.com/natalie-perlin> ? Thanks, Gillian — Reply to this email directly, view it on GitHub <#253 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQJDAWQOGPYPDHYLRW3UDMTYTI4M3AVCNFSM6AAAAABAQF5CCCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DINBTGYYTE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[natalie-perlin]

Joey,
Thank you for letting the EPIC team know it's all working good on your end.
Best regards,
Natalie Perlin/ EPIC Software Integration team