Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test #3120

Merged
merged 47 commits into from
Dec 12, 2024

Conversation

DavidNew-NOAA
Copy link
Contributor

@DavidNew-NOAA DavidNew-NOAA commented Nov 21, 2024

Description

This PR is a companion to GDASApp PR #1365 (merged).

It turns C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test using the JEDI application testing feature. This feature is turned on using the new DO_TEST_MODE parameter added to config.base in GW PR #3115. This parameter is set to "YES" in the yaml defaults for the JEDI-based CI tests in GW.

The motivation for this PR is a need to catch changes in JEDI which alter the outputs of our applications.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? YES
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES
    • EMC verif-global
    • GDAS PR #1365 (merged)
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA, C48mx500_3DVarAOWCDA, and C48mx500_hybAOWCDA have been tested successfully on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me once the change has been made to use the same testing option flag as the other PR

@DavidNew-NOAA
Copy link
Contributor Author

@CoryMartin-NOAA Done!

@DavidNew-NOAA
Copy link
Contributor Author

I misunderstood how default yamls work. I'm merging Cory's branch into mine and redoing the test references

DavidNew-NOAA added a commit to NOAA-EMC/GDASApp that referenced this pull request Nov 25, 2024
This PR is a companion to GW PR
[#3120](NOAA-EMC/global-workflow#3120).

It does a couple things:

1. 5 GW CI tests are added/extended as CTests in GDASApp, running
through to the fcst jobs in the first full-cycle. These CI tests are:
```C96C48_hybatmDA```, ```C96C48_ufs_hybatmDA```,
```C96C48_hybatmaerosnowDA```, ```C48mx500_3DVarAOWCDA```, and
```C48mx500_hybAOWCDA```.
2. Test references are added for ```C96C48_ufs_hybatmDA``` and
```C48mx500_3DVarAOWCDA```, so that we're actually testing the output.
3. These CTests are turned on by default in a workflow build, rather
than having to mess with the ```CMakeCache.txt``` file and re-running
make. This will allow us to use these tests in nightly testing.
4. ```test/gw-ci/CMakeLists.txt``` is refactored quite a bit.
5. There are 89 CTests, but for 5 CI tests, but I added task
dependencies, so they can be run in parallel.

The primary motivation for this PR is that we can run CI for our nightly
testing of GDASApp. Also, anyone with a PR can easily do CI testing
through CTests.

---------

Co-authored-by: Russ-Treadon-NOAA <[email protected]>
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Dec 11, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Dec 11, 2024
@emcbot
Copy link

emcbot commented Dec 11, 2024

Checkout Failed on Hera in Build# 2: Could not perform submodule update

1 similar comment
@emcbot
Copy link

emcbot commented Dec 11, 2024

Checkout Failed on Hera in Build# 2: Could not perform submodule update

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Dec 11, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Dec 11, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Dec 11, 2024
@emcbot
Copy link

emcbot commented Dec 11, 2024

Build FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI/3120/gfs/sorc/logs/build_gsi_utils.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Dec 11, 2024

Build FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI/3120/gefs/sorc/logs/build_gfs_utils.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Dec 11, 2024
@RussTreadon-NOAA
Copy link
Contributor

Build FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI/3120/gefs/sorc/logs/build_gfs_utils.log

Follow link here to view the contents of the above file(s): (link)

Build failed due to fork resource unavailable

make[2]: fork: Resource temporarily unavailable
make[2]: Leaving directory '/scratch1/NCEPDEV/global/CI/3120/gfs/sorc/gsi_utils.fd/build'
make[1]: *** [CMakeFiles/Makefile2:531: src/EnKF/gfs/src/calc_increment_ens.fd/CMakeFiles/calc_increment_ens.x.dir/all] Error 2
make[1]: Leaving directory '/scratch1/NCEPDEV/global/CI/3120/gfs/sorc/gsi_utils.fd/build'
make: *** [Makefile:136: all] Error 2

@RussTreadon-NOAA
Copy link
Contributor

WCOSS2 g-w CI

Install DavidNew-NOAA:feature/gw-ci at 5243826 on Dogwood. Run g-w CI with the following results

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Dec 11 2024 17:30:35    Dec 11 2024 18:50:07
202103231800        Done    Dec 11 2024 17:30:35    Dec 11 2024 19:00:16
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Dec 11 2024 17:30:37    Dec 11 2024 17:55:12
202103250000      Active    Dec 11 2024 17:30:37             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Dec 11 2024 17:30:38    Dec 11 2024 17:55:15
202103250000      Active    Dec 11 2024 17:30:38             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Dec 11 2024 17:30:43    Dec 11 2024 19:30:16
202103231800        Done    Dec 11 2024 17:30:43    Dec 11 2024 19:40:24
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Dec 11 2024 17:30:46    Dec 11 2024 18:00:27
202112210000        Done    Dec 11 2024 17:30:46    Dec 11 2024 20:20:17
202112210600        Done    Dec 11 2024 17:30:46    Dec 11 2024 20:05:18
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Dec 11 2024 17:30:48    Dec 11 2024 18:05:36
202112201800        Done    Dec 11 2024 17:30:48    Dec 11 2024 20:20:22
202112210000        Done    Dec 11 2024 17:30:48    Dec 11 2024 20:15:28
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Dec 11 2024 17:30:50    Dec 11 2024 18:00:36
202112210000        Done    Dec 11 2024 17:30:50    Dec 11 2024 20:25:27
202112210600        Done    Dec 11 2024 17:30:50    Dec 11 2024 20:20:25
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Dec 11 2024 17:30:51    Dec 11 2024 18:00:39
202402240000        Done    Dec 11 2024 17:30:51    Dec 11 2024 21:10:29
202402240600        Done    Dec 11 2024 17:30:51    Dec 11 2024 21:11:37          

All streams successfully ran to completion except except C48mx500_3DVarAOWCDA and C48mx500_hybAOWCDA.

These streams are stuck because required GDA files for 20210325 00Z do not exist on Dogwood. g-w issue #3159 has been opened requesting that the GDA directory for 20210325 00Z be copied from Hera to WCOSS2.

@RussTreadon-NOAA
Copy link
Contributor

Orion g-w CI

Install DavidNew-NOAA:feature/gw-ci at 5243826 on Orion. Run g-w CI with the following results

/work2/noaa/stmp/rtreadon/EXPDIR/C48_ATM_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 11 2024 21:11:13    Dec 11 2024 22:20:06
202103231800        Done    Dec 11 2024 21:11:13    Dec 11 2024 22:35:04

/work2/noaa/stmp/rtreadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 11 2024 21:11:15    Dec 11 2024 21:30:09
202103250000        Done    Dec 11 2024 21:11:15    Dec 11 2024 22:50:06

/work2/noaa/stmp/rtreadon/EXPDIR/C48mx500_hybAOWCDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 11 2024 21:11:16    Dec 11 2024 21:35:08
202103250000        Done    Dec 11 2024 21:11:16    Dec 11 2024 23:00:10

/work2/noaa/stmp/rtreadon/EXPDIR/C48_S2SW_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 11 2024 21:11:28    Dec 12 2024 00:46:21
202103231800        Done    Dec 11 2024 21:11:28    Dec 12 2024 01:06:10

/work2/noaa/stmp/rtreadon/EXPDIR/C96_atm3DVar_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 11 2024 21:11:30    Dec 11 2024 21:35:13
202112210000        Done    Dec 11 2024 21:11:30    Dec 12 2024 00:20:13
202112210600        Done    Dec 11 2024 21:11:30    Dec 11 2024 23:50:12

/work2/noaa/stmp/rtreadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Dec 11 2024 21:11:31    Dec 11 2024 21:35:15
202112201800        Done    Dec 11 2024 21:11:31    Dec 12 2024 00:10:15
202112210000        Done    Dec 11 2024 21:11:31    Dec 12 2024 00:10:15

/work2/noaa/stmp/rtreadon/EXPDIR/C96C48_hybatmDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 11 2024 21:11:33    Dec 11 2024 21:35:17
202112210000        Done    Dec 11 2024 21:11:33    Dec 12 2024 00:20:17
202112210600        Done    Dec 11 2024 21:11:33    Dec 12 2024 00:00:18

/work2/noaa/stmp/rtreadon/EXPDIR/C96C48_ufs_hybatmDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Dec 11 2024 21:11:35    Dec 11 2024 21:35:19
202402240000        Done    Dec 11 2024 21:11:35    Dec 12 2024 01:10:23
202402240600        Done    Dec 11 2024 21:11:35    Dec 12 2024 01:29:16

All streams successfully ran to completion.

@RussTreadon-NOAA
Copy link
Contributor

Hera g-w CI

Install DavidNew-NOAA:feature/gw-ci at 5243826 on Hera. Run g-w CI with the following results

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48_ATM_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 12 2024 11:11:02    Dec 12 2024 13:10:14
202103231800        Done    Dec 12 2024 11:11:02    Dec 12 2024 13:35:16

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48_S2SW_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 12 2024 11:11:04    Dec 12 2024 15:25:22
202103231800        Done    Dec 12 2024 11:11:04    Dec 12 2024 15:39:59

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 12 2024 11:11:13    Dec 12 2024 11:40:27
202103250000        Done    Dec 12 2024 11:11:13    Dec 12 2024 13:25:21

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_hybAOWCDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 12 2024 11:11:14    Dec 12 2024 13:47:10
202103250000        Done    Dec 12 2024 11:11:14    Dec 12 2024 16:10:34

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 12 2024 11:11:17    Dec 12 2024 11:40:28
202112210000        Done    Dec 12 2024 11:11:17    Dec 12 2024 14:40:39
202112210600        Done    Dec 12 2024 11:11:17    Dec 12 2024 14:02:23

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Dec 12 2024 11:11:18    Dec 12 2024 11:45:29
202112201800        Done    Dec 12 2024 11:11:18    Dec 12 2024 14:45:28
202112210000        Done    Dec 12 2024 11:11:18    Dec 12 2024 14:35:26

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Dec 12 2024 11:11:20    Dec 12 2024 11:40:31
202402240000        Done    Dec 12 2024 11:11:20    Dec 12 2024 15:40:11
202402240600        Done    Dec 12 2024 11:11:20    Dec 12 2024 15:35:28

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96_atm3DVar_pr3120
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 12 2024 11:11:24    Dec 12 2024 13:54:01
202112210000        Done    Dec 12 2024 11:11:24    Dec 12 2024 17:52:15
202112210600        Done    Dec 12 2024 11:11:24    Dec 12 2024 18:18:51

All streams successfully ran to completion.

Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

g-w CI has been run on WCOSS2 (Dogwood), Hera, and Orion. All jobs in the tested g-w CI streams successfully ran to completion.

Approve.

@DavidNew-NOAA
Copy link
Contributor Author

Thanks for the manual testing @RussTreadon-NOAA

@RussTreadon-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA , @aerorahul and @KateFriedman-NOAA :

g-w CI testing has been completed on WCOSS2 (Dogwood), Hera, and Orion. All tested streams successfully ran to completion with the following caveats

  1. GEFS streams C48_S2SWA_gefs and C96_S2SWA_gefs_replay_ics were not run since the default sorc/build_all.sh builds the UFS model for GFS, not GEFS, applications.
  2. C48mx500_3DVarAOWCDA and C48mx500_hybAOWCDA are unable to advance past 20210325 00Z because GDA directory /lfs/h2/emc/dump/noscrub/dump/gdas.20210325 does not exist on Dogwood. This has been reported in g-w issue GDA files for 20210325 00Z required for C48mx500_3DVarAOWCDA & C48mx500_hybAOWCDA missing on WCOSS2 #3159. Additionally, WCOSS2.env has an erroneous APRUN_OCNANALECEN. This problem has been reported in g-w issue WCOSS2.env uses invalid options for APRUN_OCNANALECEN #3158

Both C48mx500_3DVarAOWCDA and C48mx500_hybAOWCDA are set to be skipped on WCOSS2. I run AOWCDA on WCOSS2 because GFS v17 will exercise AOWCDA.

Is the amount of g-w CI tested reported in this issue sufficient for this PR to be approved and merged into develop?

If not, what else must done?

@RussTreadon-NOAA RussTreadon-NOAA added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Dec 12, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 0c7a3c6 into NOAA-EMC:develop Dec 12, 2024
9 of 11 checks passed
@RussTreadon-NOAA
Copy link
Contributor

Thank you @WalterKolczynski-NOAA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants