Runtime error with GNU libraries for scream tests

[ndkeen]

For test like: SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1.pm-cpu_gnu

I get:

21: /pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/nup/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1.pm-cpu_gnu.20241010_144522_00l7iv/bld/e3sm.exe: /opt/cray/pe/gcc-libs/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /pscratch\
/sd/n/ndk/e3sm_scratch/pm-cpu/nup/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1.pm-cpu_gnu.20241010_144522_00l7iv/bld/e3sm.exe)

where "normal" non-scream cases are ok with gnu, like
SMS_D_Ln5.ne4pg2_oQU480.F2010.pm-cpu_gnu

Could it be related to E3SM-Project/scream#2972 ?

[xylar]

@ndkeen, I've seen something similar (see E3SM-Ocean-Discussion#112) when I try to build E3SM tests (even non-scream tests) on LANL's Chicoma machine, which is pretty similar to Perlmutter. It looked to me like Chicoma has a outdated symlink at /opt/cray/pe/gcc-libs/libstdc++.so.6 to an older version of gcc, which seemed to explain the missing symbols for newer gcc (see https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html). I haven't had a chance to check on Perlmutter but I suspect something similar is happening. Not sure why it's only triggered by scream tests there but maybe they're using newer features that require the newer GLIBCXX version?

It seems like the rpath for /opt/cray/pe/gcc-libs/ is hard-coded into ftn and the other cray compiler wrappers. So it seems like there's an expectation that the symlinks in that location are maybe the latest versions that are supported (since different gcc versions get the same rpath).