Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 #3106

DavidBurrows-NCO · 2024-11-15T16:31:08Z

Description

What:
Correct build/run for C48_ATM and C48_S2SW on Gaea C5. Add build and run capability for C48_ATM, C48_S2SW, and C96_atm3DVar on Gaea C6.
Why:
After the C5 OS upgrade, submodules no longer built in the global-workflow. This PR correct that and adds build/run capability to C6.

Resolves #3011
Depends on:
ufs-community/ufs-weather-model#2448
ufs-community/UFS_UTILS#995
NOAA-EMC/gfs-utils#87
NOAA-EMC/UPP#1070
NOAA-EMC/GSI#800
NOAA-EMC/GSI-utils#55
NOAA-EMC/GSI-Monitor#146
NOAA-EMC/GDASApp#1361

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS Update build scripts on Gaea-C5 to conform with ufs-wx-model and global-workflow GDASApp#1361
- GFS-utils Add/fix build capability for Gaea-C5, Gaea-C6, and container gfs-utils#87
- GSI Add/fix build capability for Gaea-C5, Gaea-C6, and container GSI#800
- GSI-monitor Add/fix build capability for Gaea-C5, Gaea-C6, and container GSI-Monitor#146
- GSI-utils Add/fix build capability for Gaea-C5, Gaea-C6, and container GSI-utils#55
- UFS-utils Update scripts to build on both Gaea-C5 and C6 ufs-community/UFS_UTILS#995
- UFS-weather-model Gaea C6 support for UFSWM ufs-community/ufs-weather-model#2448
- wxflow

How has this been tested?

C5 and C6: clone, built, and ran C48_ATM and C48_S2SW successfully.
C96_atm3DVar is hanging in sfcanl jobs.

Checklist

DavidBurrows-NCO · 2024-11-15T16:44:04Z

Hi @aerorahul @WalterKolczynski-NOAA We're still waiting on build merges for some submodules, so I've left this PR in draft. From our conversation Tuesday, I've pointed the submodules that were merged to their respective head of develop and the others to my commit for now. Should I be pointing to my submodule commits instead to limit the number of changes coming into GW? Thanks

jswhit · 2024-11-15T18:41:19Z

sorc/build_all.sh needs the following update:

--- sorc/build_all.sh
+++ sorc/build_all.sh
@@ -149,7 +149,7 @@ build_opts["ww3prepost"]="${_wave_opt} ${_verbose_opt} ${_build_ufs_opt} ${_buil

 # Optional DA builds
 if [[ "${_build_ufsda}" == "YES" ]]; then
-   if [[ "${MACHINE_ID}" != "orion" && "${MACHINE_ID}" != "hera" && "${MACHINE_ID}" != "hercules" && "${MACHINE_ID}" != "wcoss2" && "${MACHINE_ID}" != "noaacloud" && "${MACHINE_ID}" != "gaea" ]]; then
+   if [[ "${MACHINE_ID}" != "orion" && "${MACHINE_ID}" != "hera" && "${MACHINE_ID}" != "hercules" && "${MACHINE_ID}" != "wcoss2" && "${MACHINE_ID}" != "noaacloud" && "${MACHINE_ID}" != "gaeac5" && "${MACHINE_ID}" != "gaeac6" ]]; then
       echo "NOTE: The GDAS App is not supported on ${MACHINE_ID}.  Disabling build."
    else
       build_jobs["gdas"]=8

jswhit · 2024-11-15T18:45:24Z

also ush/load_ufsda_modules.sh needs

--- a/ush/load_ufsda_modules.sh
+++ b/ush/load_ufsda_modules.sh
@@ -34,13 +34,13 @@ source "${HOMEgfs}/ush/module-setup.sh"
 module use "${HOMEgfs}/sorc/gdas.cd/modulefiles"

 case "${MACHINE_ID}" in
-  ("hera" | "orion" | "hercules" | "wcoss2")
+  ("hera" | "orion" | "hercules" | "gaeac5" | "gaeac6" | "wcoss2")
     module load "${MODS}/${MACHINE_ID}"
     ncdump=$( command -v ncdump )
     NETCDF=$( echo "${ncdump}" | cut -d " " -f 3 )
     export NETCDF
     ;;
-  ("jet" | "gaea" | "s4" | "acorn")
+  ("jet" | "s4" | "acorn")
     echo WARNING: UFSDA NOT SUPPORTED ON THIS PLATFORM
     ;;
   *)

DavidBurrows-NCO · 2024-11-15T18:54:35Z

also ush/load_ufsda_modules.sh needs

Thanks @jswhit I pushed changes to ush/load_ufsda_modules.sh and sorc/build_all.sh

jswhit · 2024-11-15T19:16:55Z

also...

workflow/hosts/gaeac6.yaml and gaeac5.yaml:

-QUEUE_SERVICE: normal
+QUEUE_SERVICE: hpss
 PARTITION_BATCH: batch
-PARTITION_SERVICE: batch
+PARTITION_SERVICE: dtn_f5_f6

and modulefiles/module_gwsetup.gaeac6.lua:

-prepend_path("MODULEPATH", "/ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")

jswhit · 2024-11-15T19:37:38Z

env/GAEAC5.env and GAEAC6.env seem to be missing a bunch of stuff. I just copied HERCULES.env for both, and made some minor mods (see https://github.com/jswhit2/global-workflow/blob/develop/env/GAEAC5.env)

jswhit · 2024-11-15T20:03:14Z

build_ww3prepost is failing for me on both c5 and c6 (using ufs-wx-model 2448)

DavidBurrows-NCO · 2024-11-15T20:03:31Z

missing a bunch of stuff

@jswhit It's not really missing but intentionally minimized at the request of EMC porting to a new machine. Instead, we started from a nearly blank canvas and have been building up. Currently, the C5 and C6.env files are set up for C48_ATM, C48_S2SW, and C96_atm3DVar jobs. The 3DVarAOWCDA configuration you're running will definitely have some additional jobs. If you send those particular job names (or "step" in the env file). I will add them to the files.

JessicaMeixner-NOAA · 2024-11-18T21:09:36Z

build_ww3prepost is failing for me on both c5 and c6 (using ufs-wx-model 2448)

@jswhit - can you point me to a log file? Maybe I can look and see if something is easy to fix with this.

jswhit · 2024-11-18T21:29:05Z

build_ww3prepost is failing for me on both c5 and c6 (using ufs-wx-model 2448)

@jswhit - can you point me to a log file? Maybe I can look and see if something is easy to fix with this.

@JessicaMeixner-NOAA here is the error:

gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(451): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [WAV_RESTART_MOD]    use wav_restart_mod, only : read_restart
--------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(975): error #6632: Keyword arguments are invalid without an explicit interface.   [VA]            call read_restart(trim(fname), va=va, mapsta=mapsta, mapst2=mapst2)
-------------------------------------------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(975): error #6632: Keyword arguments are invalid without an explicit interface.   [MAPSTA]
            call read_restart(trim(fname), va=va, mapsta=mapsta, mapst2=mapst2)
--------------------------------------------------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(975): error #6632: Keyword arguments are invalid without an explicit interface.   [MAPST2]
            call read_restart(trim(fname), va=va, mapsta=mapsta, mapst2=mapst2)
-----------------------------------------------------------------^
/gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90(451): error #6580: Name in only-list does not exist or is not accessible.   [READ_RESTART]
    use wav_restart_mod, only : read_restart
--------------------------------^
compilation aborted for /gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/global-workflow-jswhit2/sorc/ufs_model.fd/WW3/model/src/w3initmd.F90 (code 1)

JessicaMeixner-NOAA · 2024-11-18T22:05:43Z

@jswhit - Okay I know what the issue is, but it'll take a minute to get it fixed. The issue crept in with ufs-community/ufs-weather-model#2445 and we didn't catch it. If you go back one-commit of ufs-waether-model, hopefully things will run. We'll get a fix in as soon as possible.

jswhit · 2024-11-19T22:53:59Z

@JessicaMeixner-NOAA I'm seeing this error in the gdas_fcst step on c6 when I run with ufs-wx-model 2448

424:  (abort_ice)ABORTED:
424:  (abort_ice) error =
424:  (construct_filename) ERROR: history filename already used for another history s
424:  tream iceh_inst.2021-03-24-10800.nc

and the traceback looks like this

473: ufs_model.x        0000000005E9CD8B  ice_broadcast_mp_         252  ice_broadcast.F90
473: ufs_model.x        0000000005F055E3  ice_history_write         169  ice_history_write.F90
473: ufs_model.x        0000000005C2A4E2  ice_history_mp_ac        4134  ice_history.F90
473: ufs_model.x        0000000005EE77FC  cice_runmod_mp_ci         367  CICE_RunMod.F90
473: ufs_model.x        0000000005B7DA06  ice_comp_nuopc_mp        1204  ice_comp_nuopc.F90
473: ufs_model.x        0000000000D05438  Unknown               Unknown  Unknown

Do you know of any recenter cice changes that could cause this?

JessicaMeixner-NOAA · 2024-11-19T22:59:43Z

I don't know but I'm not as caught up on all the recent ufs wm changes as I normally am, but taking a quick look at ufs-weather-model says CICE hasn't been updated in 2 months.

jswhit · 2024-11-19T23:54:09Z

For some more context on the cice error, from ice_diag.d:

(ice_comp_nuopc):(ModelAdvance) cice istep, nextsw_cday =         15      0.83111111111111D+02
 (ice_pio_init) create file ./CICE_OUTPUT/iceh_inst.2021-03-24-09600.nc

 Finished writing ./CICE_OUTPUT/iceh_inst.2021-03-24-09600.nc
(ice_comp_nuopc):(ModelAdvance) cice istep, nextsw_cday =         16      0.83118055555556D+02
 (ice_pio_init) create file ./CICE_OUTPUT/iceh_inst.2021-03-24-10200.nc

 Finished writing ./CICE_OUTPUT/iceh_inst.2021-03-24-10200.nc
(ice_comp_nuopc):(ModelAdvance) cice istep, nextsw_cday =         17      0.83125000000000D+02
 (ice_pio_init) create file ./CICE_OUTPUT/iceh_inst.2021-03-24-10800.nc

 Finished writing ./CICE_OUTPUT/iceh_inst.2021-03-24-10800.nc
 (construct_filename) history stream =            4
 (construct_filename) history filename = iceh_inst.2021-03-24-10800.nc
 (construct_filename) filename in use for stream            3
 (construct_filename) filename for stream iceh_inst.2021-03-24-10800.nc
 (construct_filename) Use namelist hist_suffix so history filenames are unique

jswhit2 · 2024-11-21T21:15:10Z

The problem with the ice model (and a potential fix) are documented in PR #3121

JessicaMeixner-NOAA · 2025-01-17T14:59:42Z

@DavidBurrows-NCO - If you will merge in develop, it will use a ufs-weather-model hash beyond what you need for Gaea and should take care of the model related issues with RESTART_FH.

aerorahul · 2025-01-21T15:05:56Z

@DavidBurrows-NCO
Can you merge in develop and resolve the conflicts. With that, we should be able to move forward on this PR.

DavidBurrows-NCO · 2025-01-21T15:22:45Z

Thanks @JessicaMeixner-NOAA for reaching out about this. @aerorahul Yes..working to update my branch to develop today and will retest C48_S2SW.

DavidBurrows-NCO · 2025-01-21T20:36:15Z

@aerorahul I aligned my branch with develop today and tested C48_S2SW on C6 successfully. F5 is unmounted from C5 today, so I will have to test C5 tomorrow.

DavidBurrows-NCO · 2025-01-22T16:22:17Z

Morning @aerorahul. I was able to clone, build, and run C48_S2SW on C5 this morning. The only issue I see is when create_experiment runs on C5 and C6, memory is set in the xml file for the gfs_wavepostsbs job. This causes a failure in job submission. I'm not sure where that is occurring though.

aerorahul

looks good to me.
Since this updates a lot of submodules, it should be tested across all machines.

…ow into gw_c5OSc6

aerorahul

looks good.
Since the PR does not update any submodules or impacts running on any of the machines WCOSS2, Hera, Hercules, or Orion, there is no need to run CI on this PR.

KateFriedman-NOAA

Looks good, thanks @DavidBurrows-NCO !

ush/load_ufsda_modules.sh

aerorahul

lgtm.

* develop: Only run METplus in the 3Dvar tests (NOAA-EMC#3245) Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 (NOAA-EMC#3106)

DavidBurrows-NCO and others added 7 commits October 25, 2024 07:05

Updates for setup_expt and setup_xml workflow on Gaea C5 and C6

e97d26d

Merge branch 'NOAA-EMC:develop' into gw_c5OSc6

ecb706c

update submodules and some files to run on C5/C6

d7e2398

modulefile updates for C6

3dbc625

merge conflicts

a8d2720

module file updates

296b5c3

update C5 resources

16d1ff4

enable C5 and C6 build_all and load modules

5973d8c

update ufs_utils hash to develop

3c946e3

remove data in path for C5 fix files to be consistent with other systems

fc56219

Merge branch 'NOAA-EMC:develop' into gw_c5OSc6

ad28704

This was referenced Nov 18, 2024

WW3 cannot build pre-and-post processing jobs ufs-community/ufs-weather-model#2502

Closed

Issue building pre/post wave jobs with latest ufs-weather-model #3110

Closed

merging gdas.cd conflict

7e4e168

jswhit2 mentioned this pull request Nov 21, 2024

Enable DA cycling on gaea #3032

Open

10 tasks

Merge branch 'NOAA-EMC:develop' into gw_c5OSc6

2ae073b

aerorahul removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Jan 21, 2025

DavidBurrows-NCO and others added 2 commits January 21, 2025 12:31

updating conflicts and submods

97c8936

Merge branch 'NOAA-EMC:develop' into gw_c5OSc6

9c043e4

Merge branch 'develop' into gw_c5OSc6

7e81dc7

aerorahul previously approved these changes Jan 22, 2025

View reviewed changes

DavidBurrows-NCO added 2 commits January 22, 2025 11:45

update submodule hashes

78c4e08

Merge branch 'gw_c5OSc6' of github.com:DavidBurrows-NCO/global-workfl…

387e20f

…ow into gw_c5OSc6

DavidBurrows-NCO dismissed aerorahul’s stale review via 387e20f January 22, 2025 16:51

aerorahul previously approved these changes Jan 22, 2025

View reviewed changes

aerorahul requested review from KateFriedman-NOAA and WalterKolczynski-NOAA January 22, 2025 19:56

KateFriedman-NOAA previously approved these changes Jan 22, 2025

View reviewed changes

aerorahul reviewed Jan 22, 2025

View reviewed changes

ush/load_ufsda_modules.sh Show resolved Hide resolved

add gaeac5/c6 to module loads for da

ab93c5f

DavidBurrows-NCO dismissed stale reviews from KateFriedman-NOAA and aerorahul via ab93c5f January 22, 2025 20:07

aerorahul requested review from aerorahul and KateFriedman-NOAA January 22, 2025 21:31

aerorahul approved these changes Jan 22, 2025

View reviewed changes

aerorahul merged commit 01f9c35 into NOAA-EMC:develop Jan 22, 2025
5 checks passed

DavidBurrows-NCO deleted the gw_c5OSc6 branch January 23, 2025 13:39

tsga added a commit to tsga/global-workflow that referenced this pull request Jan 23, 2025

Merge branch 'develop' into feature/lnd_regrid_iau

a10f618

* develop: Only run METplus in the 3Dvar tests (NOAA-EMC#3245) Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 (NOAA-EMC#3106)

tsga added a commit to tsga/global-workflow that referenced this pull request Jan 23, 2025

Merge branch 'develop' into feature/soilanal_det

7577ebb

* develop: Only run METplus in the 3Dvar tests (NOAA-EMC#3245) Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 (NOAA-EMC#3106)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 #3106

Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 #3106

DavidBurrows-NCO commented Nov 15, 2024 •

edited

Loading

DavidBurrows-NCO commented Nov 15, 2024

jswhit commented Nov 15, 2024

jswhit commented Nov 15, 2024

DavidBurrows-NCO commented Nov 15, 2024

jswhit commented Nov 15, 2024 •

edited

Loading

jswhit commented Nov 15, 2024

jswhit commented Nov 15, 2024 •

edited

Loading

DavidBurrows-NCO commented Nov 15, 2024

JessicaMeixner-NOAA commented Nov 18, 2024

jswhit commented Nov 18, 2024

JessicaMeixner-NOAA commented Nov 18, 2024

jswhit commented Nov 19, 2024 •

edited

Loading

JessicaMeixner-NOAA commented Nov 19, 2024

jswhit commented Nov 19, 2024

jswhit2 commented Nov 21, 2024

JessicaMeixner-NOAA commented Jan 17, 2025

aerorahul commented Jan 21, 2025

DavidBurrows-NCO commented Jan 21, 2025

DavidBurrows-NCO commented Jan 21, 2025

DavidBurrows-NCO commented Jan 22, 2025

aerorahul left a comment

aerorahul left a comment

KateFriedman-NOAA left a comment

aerorahul left a comment

Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 #3106

Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 #3106

Conversation

DavidBurrows-NCO commented Nov 15, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

DavidBurrows-NCO commented Nov 15, 2024

jswhit commented Nov 15, 2024

jswhit commented Nov 15, 2024

DavidBurrows-NCO commented Nov 15, 2024

jswhit commented Nov 15, 2024 • edited Loading

jswhit commented Nov 15, 2024

jswhit commented Nov 15, 2024 • edited Loading

DavidBurrows-NCO commented Nov 15, 2024

JessicaMeixner-NOAA commented Nov 18, 2024

jswhit commented Nov 18, 2024

JessicaMeixner-NOAA commented Nov 18, 2024

jswhit commented Nov 19, 2024 • edited Loading

JessicaMeixner-NOAA commented Nov 19, 2024

jswhit commented Nov 19, 2024

jswhit2 commented Nov 21, 2024

JessicaMeixner-NOAA commented Jan 17, 2025

aerorahul commented Jan 21, 2025

DavidBurrows-NCO commented Jan 21, 2025

DavidBurrows-NCO commented Jan 21, 2025

DavidBurrows-NCO commented Jan 22, 2025

aerorahul left a comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

DavidBurrows-NCO commented Nov 15, 2024 •

edited

Loading

jswhit commented Nov 15, 2024 •

edited

Loading

jswhit commented Nov 15, 2024 •

edited

Loading

jswhit commented Nov 19, 2024 •

edited

Loading