troubleshooting ufs weather model failure

[jkbk2004]

The ufs weather model cpld_control_p8 (S2SWA) fails when built/run with spack-stack/1.3.1. Experiment setup is available on Cheyenne and Orion. Detail information is available at ufs-community/ufs-weather-model#1737. Can gocart team take a look at the setup on Orion?

[jkbk2004]

@weiyuan-jiang @mathomp4 Are you still accessible to Orion?

[mathomp4]

@jkbk2004 I have access, but @weiyuan-jiang is the one that mainly runs out there. @aoloso has as well. I'll ping them.

[junwang-noaa]

We had some discussion last year on this problem. Please see GOCART issue #127 .
My understanding is that the code updates were committed to GOCART and MAPL, but we need gnu 11.3 to run the UFS aerosol tests.

[weiyuan-jiang]

I can confirm those changes have been made into MAPL and GOCART. I would suggest to upgrade gnu and mpi stack first. It would be very hard to go back to debug the codes with the old version of gnu and mpi.

[ulmononian]

@junwang-noaa @weiyuan-jiang thanks for this tip. i looked over #127, and as @weiyuan-jiang mentioned, it seems that the fix for the errors in that issue was merged several version of mapl ago. that said, we can run w/ aerosols (e.g., cpld_control_p8) on hera with gnu/9.2.0 and openmpi/4.1.5 with spack-stack/1.3.1., but it does not work on cheyenne ( gnu/10.1.0 / openmpi/4.1.1 ) or orion (gnu/10.2.0 / openmpi/4.0.4) (as described in UFS-WM #1737).

is the suggestion to use a newer version of gnu on cheyenne/orion? cheynne has 11.1.0, 11.2.0, & 12.1.0, while the newest available on orion is 11.3.0.

thanks!

[jkbk2004]

Clearly, it will be worth to try gnu11.3 on orion. Sounds like I still keep my old folder: /work/noaa/epic-ps/jongkim/UFS-RT-tests/rt-gnu11.3.

[ulmononian]

we will have to rebuild the entire spack-stack w/ newer gnu versions of both orion and cheyenne to test this, so it will take some time.

[mark-a-potts]

I am not sure that is worth @ulmononian 's time to rebuild for this issue.

[mathomp4]

I think we have also seen issues with MPI_Win_Fence (which I think I saw in the thread?) when using Open MPI 4.0. The solution then was to move to Open MPI 4.1.

Note that here at GMAO, we mainly use GCC 12.1 and Open MPI 4.1.3 (I think), though I've tested up to Open MPI 4.1.5 with no issue. I am pretty sure GEOS/MAPL only likes GCC 11.3+ and 12.1+. It does not like GCC 13 at the moment. We are looking into that.

[climbfuji]

I just wanted to let you know that switching to an entirely new compiler, especially a major GNU compiler, is one of the very few things that is more involved with spack-stack. You can read all the glorious details about it here: JCSDA/spack-stack#99.

That said, it's not impossible, we can certainly do it, but we should consider other options first. Given what @mathomp4 said it seems likely that a simple update of openmpi might suffice solving the problem.

In general, I would also suggest that as the UFS transitions to spack-stack, there needs to be a communication in advance of software updates. In other words, before changes are made that require major updates of the software stack, this needs to be communicated and planned together. For example, it may be a bit of a stretch to expect that every system and every user has gcc 11 on their system. It's a bit of a pain to get gcc 11 on older Ubuntu 20 and Red Hat/CenOS 7 systems. These things need to be considered before making such updates imo.

[ulmononian]

thanks for your input, @mathomp4 & @climbfuji . based on what @climbfuji explained regarding moving to new, major gnu version, i would opt to test with a newer openmpi. however, with gnu/10.2.0 on orion, the most recent openmpi available is 4.0.5, which seems too old based on @mathomp4's MPI_Win_Fence comment.

further, on cheyenne (the primary machine where this really is a bottleneck), we have already tested with openmpi/4.1.1 (and gnu/10.1.0) without success. we could test w/ open/4.1.4 there and keep the gnu version fixed, but that's about the last option i think we'd have before running into the gnu upgrade issue.

[climbfuji]

I can build openmpi 4.1.5 on cheyenne

[climbfuji]

Let's test with the existing openmpi/4.1.4 first before I go all the way to build 4.1.5.

[climbfuji]

I am building unified-env for gnu-10.1.0 and openmpi-4.1.4 now. Will let you know when it is finished (tmw morning).

[ulmononian]

thanks @climbfuji. once it's done, i will run the wm tests.

[climbfuji]

Please give this a try:

module purge
export LMOD_TMOD_FIND_FIRST=yes
module use /glade/work/jedipara/cheyenne/spack-stack/modulefiles/misc
module load miniconda/3.9.12
module load ecflow/5.8.4
module load mysql/8.0.31

module use /glade/work/epicufsrt/contrib/spack-stack/spack-stack-1.3.1/envs/unified-env-gnu-10.1.0-openmpi-4.1.4/install/modulefiles/Core
module load stack-gcc/10.1.0
module load stack-openmpi/4.1.4
module load stack-python/3.9.12

Notes:

1. I have done zero testing with this stack
2. The default ESMF and MAPL versions may not be what you want. Please check and load the correct ones manually if needed.

[ulmononian]

thank you @climbfuji for taking the time to install the stack w/ openmpi/4.1.4. unfortunately, my initial cpld_control_p8 test on cheyenne yields this in theerr file:

+ mpirun -n 360 -N 18 ./fv3.exe
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      r14i6n18
  Framework: pml
--------------------------------------------------------------------------
[r14i6n18:64364] PML ucx cannot be selected
[r14i6n24:04294] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198

i'm looking around and it seems this is a relatively common issue w/ some openmpi versions, but have not yet found a fix.

the expdir is here: /glade/scratch/bcameron/FV3_RT/rt_37524/cpld_control_p8.

[climbfuji]

Since this openmpi was built by the sysadmins from CISL, it would be good to reach out and ask them?

Otherwise I can build 4.1.5 myself next week.

[ulmononian]

Since this openmpi was built by the sysadmins from CISL, it would be good to reach out and ask them?

Otherwise I can build 4.1.5 myself next week.

i added --mca pml ^ucx to the mpirun command based on what i read here: open-mpi/ompi#1669. this allowed the model to run (i.e. avoid the PML ucx failure), but sadly, it ran for a while and ended up failing at the same place it had been with openmpi/4.1.1., namely:

       Type 8 : Restart files second request
      -----------------------------------------
            From     : 2021/03/22 06:00:00 UTC
            To       : 2021/03/23 06:00:00 UTC
            Interval :            03:00:00

       Wave model ...
 mesh file for mom6 domain is mesh.mx100.nc
 fcst_realize, cpl_grid_id=           1
 zeroing coupling accumulated fields at kdt=            1
 zeroing coupling accumulated fields at kdt=            1

perhaps we need to try openmpi/4.1.5?