Update hercules module file to spack-stack 1.7

[guillaumevernieres]

In case somebody feels like testing this ...
Draft because it's untested and still building.

• fixes Switch to spack-stack 1.7 on Hercules #1329

[CoryMartin-NOAA]

did the github actions default image change again? all of the actions are failing...

[RussTreadon-NOAA]

did the github actions default image change again? all of the actions are failing...

Something changed on their side. Do we need to change something on our side?

[CoryMartin-NOAA]

@guillaumevernieres does this work on hercules?

[guillaumevernieres]

@guillaumevernieres does this work on hercules?

@CoryMartin-NOAA , a few things were failing, I forgot exactly what but it wasn't related to the switch to 1.7 as far as I remember.

[RussTreadon-NOAA]

Hercules ctest
Install feature/spack-stack-1.7 at 34ba0aa in g-w develop at ac3cde5. Execute test_gdasapp with following results

Test project /work/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/build
      Start 1579: test_gdasapp_util_coding_norms
 1/46 Test #1579: test_gdasapp_util_coding_norms ......................................   Passed    8.61 sec
      Start 1580: test_gdasapp_util_ioda_example
 2/46 Test #1580: test_gdasapp_util_ioda_example ......................................   Passed    8.17 sec
      Start 1581: test_gdasapp_util_prepdata
 3/46 Test #1581: test_gdasapp_util_prepdata ..........................................   Passed    6.77 sec
      Start 1582: test_gdasapp_util_rads2ioda
 4/46 Test #1582: test_gdasapp_util_rads2ioda .........................................   Passed    2.13 sec
      Start 1583: test_gdasapp_util_ghrsst2ioda
 5/46 Test #1583: test_gdasapp_util_ghrsst2ioda .......................................   Passed    0.11 sec
      Start 1584: test_gdasapp_util_rtofstmp
 6/46 Test #1584: test_gdasapp_util_rtofstmp ..........................................   Passed    1.50 sec
      Start 1585: test_gdasapp_util_rtofssal
 7/46 Test #1585: test_gdasapp_util_rtofssal ..........................................   Passed    0.41 sec
      Start 1586: test_gdasapp_util_smap2ioda
 8/46 Test #1586: test_gdasapp_util_smap2ioda .........................................   Passed    0.10 sec
      Start 1587: test_gdasapp_util_smos2ioda
 9/46 Test #1587: test_gdasapp_util_smos2ioda .........................................   Passed    0.14 sec
      Start 1588: test_gdasapp_util_viirsaod2ioda
10/46 Test #1588: test_gdasapp_util_viirsaod2ioda .....................................   Passed    0.10 sec
      Start 1589: test_gdasapp_util_icecamsr2ioda
11/46 Test #1589: test_gdasapp_util_icecamsr2ioda .....................................   Passed    0.09 sec
      Start 1590: test_gdasapp_util_icecmirs2ioda
12/46 Test #1590: test_gdasapp_util_icecmirs2ioda .....................................   Passed    0.10 sec
      Start 1591: test_gdasapp_util_icecjpssrr2ioda
13/46 Test #1591: test_gdasapp_util_icecjpssrr2ioda ...................................   Passed    0.10 sec
      Start 1942: test_gdasapp_check_python_norms
14/46 Test #1942: test_gdasapp_check_python_norms .....................................   Passed    6.64 sec
      Start 1943: test_gdasapp_check_yaml_keys
15/46 Test #1943: test_gdasapp_check_yaml_keys ........................................   Passed    1.44 sec
      Start 1944: test_gdasapp_jedi_increment_to_fv3
16/46 Test #1944: test_gdasapp_jedi_increment_to_fv3 ..................................   Passed    4.17 sec
      Start 1945: test_gdasapp_fv3jedi_fv3inc
17/46 Test #1945: test_gdasapp_fv3jedi_fv3inc .........................................   Passed   43.29 sec
      Start 1946: test_gdasapp_snow_create_ens
18/46 Test #1946: test_gdasapp_snow_create_ens ........................................   Passed    1.02 sec
      Start 1947: test_gdasapp_snow_imsproc
19/46 Test #1947: test_gdasapp_snow_imsproc ...........................................   Passed    4.31 sec
      Start 1948: test_gdasapp_snow_apply_jediincr
20/46 Test #1948: test_gdasapp_snow_apply_jediincr ....................................   Passed    9.58 sec
      Start 1949: test_gdasapp_snow_letkfoi_snowda
21/46 Test #1949: test_gdasapp_snow_letkfoi_snowda ....................................   Passed   32.56 sec
      Start 1950: test_gdasapp_convert_bufr_adpsfc_snow
22/46 Test #1950: test_gdasapp_convert_bufr_adpsfc_snow ...............................   Passed    2.80 sec
      Start 1951: test_gdasapp_WCDA-3DVAR-C48mx500
23/46 Test #1951: test_gdasapp_WCDA-3DVAR-C48mx500 ....................................   Passed   21.82 sec
      Start 1952: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_stage_ic_202103241200
24/46 Test #1952: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_stage_ic_202103241200 .........   Passed  110.86 sec
      Start 1953: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_fcst_202103241200
25/46 Test #1953: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_fcst_202103241200 .............   Passed  566.67 sec
      Start 1954: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_prepoceanobs_202103241800
26/46 Test #1954: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_prepoceanobs_202103241800 .....   Passed  287.82 sec
      Start 1955: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marinebmat_202103241800
27/46 Test #1955: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marinebmat_202103241800 .......   Passed  224.03 sec
      Start 1956: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlinit_202103241800
28/46 Test #1956: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlinit_202103241800 ....***Failed  211.23 sec
      Start 1957: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlvar_202103241800
29/46 Test #1957: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlvar_202103241800 .....***Failed   77.13 sec
      Start 1958: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlchkpt_202103241800
30/46 Test #1958: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlchkpt_202103241800 ...***Failed  131.71 sec
      Start 1959: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlfinal_202103241800
31/46 Test #1959: test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlfinal_202103241800 ...***Failed  108.24 sec
      Start 1960: test_gdasapp_convert_bufr_adpsfc
32/46 Test #1960: test_gdasapp_convert_bufr_adpsfc ....................................   Passed   10.89 sec
      Start 1961: test_gdasapp_convert_gsi_satbias
33/46 Test #1961: test_gdasapp_convert_gsi_satbias ....................................   Passed    6.11 sec
      Start 1962: test_gdasapp_setup_atm_cycled_exp
34/46 Test #1962: test_gdasapp_setup_atm_cycled_exp ...................................   Passed    6.06 sec
      Start 1963: test_gdasapp_atm_jjob_var_init
35/46 Test #1963: test_gdasapp_atm_jjob_var_init ......................................***Failed   76.37 sec
      Start 1964: test_gdasapp_atm_jjob_var_run
36/46 Test #1964: test_gdasapp_atm_jjob_var_run .......................................***Failed   42.15 sec
      Start 1965: test_gdasapp_atm_jjob_var_inc
37/46 Test #1965: test_gdasapp_atm_jjob_var_inc .......................................***Failed   74.15 sec
      Start 1966: test_gdasapp_atm_jjob_var_final
38/46 Test #1966: test_gdasapp_atm_jjob_var_final .....................................***Failed  234.17 sec
      Start 1967: test_gdasapp_atm_jjob_ens_init
39/46 Test #1967: test_gdasapp_atm_jjob_ens_init ......................................   Passed   44.56 sec
      Start 1968: test_gdasapp_atm_jjob_ens_letkf
40/46 Test #1968: test_gdasapp_atm_jjob_ens_letkf .....................................   Passed  554.23 sec
      Start 1969: test_gdasapp_atm_jjob_ens_init_split
41/46 Test #1969: test_gdasapp_atm_jjob_ens_init_split ................................   Passed   45.96 sec
      Start 1970: test_gdasapp_atm_jjob_ens_obs
42/46 Test #1970: test_gdasapp_atm_jjob_ens_obs .......................................   Passed   42.25 sec
      Start 1971: test_gdasapp_atm_jjob_ens_sol
43/46 Test #1971: test_gdasapp_atm_jjob_ens_sol .......................................***Failed   10.26 sec
      Start 1972: test_gdasapp_atm_jjob_ens_inc
44/46 Test #1972: test_gdasapp_atm_jjob_ens_inc .......................................***Failed   42.20 sec
      Start 1973: test_gdasapp_atm_jjob_ens_final
45/46 Test #1973: test_gdasapp_atm_jjob_ens_final .....................................***Failed   42.16 sec
      Start 1974: test_gdasapp_aero_gen_3dvar_yaml
46/46 Test #1974: test_gdasapp_aero_gen_3dvar_yaml ....................................   Passed    4.66 sec

76% tests passed, 11 tests failed out of 46

Label Time Summary:
gdas-utils    =  28.34 sec*proc (13 tests)
manual        = 1739.51 sec*proc (9 tests)
script        =  28.34 sec*proc (13 tests)

Total Test time (real) = 3110.85 sec

The following tests FAILED:
        1956 - test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlinit_202103241800 (Failed)
        1957 - test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlvar_202103241800 (Failed)
        1958 - test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlchkpt_202103241800 (Failed)
        1959 - test_gdasapp_WCDA-3DVAR-C48mx500_gdas_marineanlfinal_202103241800 (Failed)
        1963 - test_gdasapp_atm_jjob_var_init (Failed)
        1964 - test_gdasapp_atm_jjob_var_run (Failed)
        1965 - test_gdasapp_atm_jjob_var_inc (Failed)
        1966 - test_gdasapp_atm_jjob_var_final (Failed)
        1971 - test_gdasapp_atm_jjob_ens_sol (Failed)
        1972 - test_gdasapp_atm_jjob_ens_inc (Failed)
        1973 - test_gdasapp_atm_jjob_ens_final (Failed)

[RussTreadon-NOAA]

test_gdasapp_atm_jjob_var_init and test_gdasapp_atm_jjob_ens_sol fail because feature/spack-stack-1.7 at 34ba0aa includes GDASApp PR #1312.

PR #1312 replaced

• replaces JCB_ALGO_YAML with JCB_ALGO_YAML_VAR in config.atmanl
• adds JCB_ALGO_YAML_SOL to config.atmensanl

These GDASApp changes require corresponding g-w changes. @DavidNew-NOAA has these changes in g-w PR #2992. I built feature/spack-stack-1.7 inside the current head of g-w develop. g-w develop does not yet contain the above JCB_ALGO changes now in feature/spack-stack-1.7.

@DavidNew-NOAA notes that ctests will fail until g-w PR #2992 is merged into g-w develop.

The remaining test_gdasapp_atm_jjob tests failed because they require test_gdasapp_atm_jjob_var_init and test_gdasapp_atm_jjob_ens_sol to successfully run.

[apchoiCMD]

Please fix the marine DA that you found out-

[guillaumevernieres]

Please fix the marine DA that you found out-

yes sir! 🫡

[guillaumevernieres]

OK, after a bit of confusion with all the parallel development that is going on with different combinations of dozens of repositories, this PR should ignore the marine DA testing. The marine DA will be broken until NOAA-EMC/global-workflow#2978 is merged.

From @RussTreadon-NOAA 's log, :

ValueError: In template_string_jinja2 the output string still contains template directives. # Set the model component to be used in later configuration

the rendered yaml still has the undefined MOM6_LEVS templated variable:

  background error:


    covariance model: SABER
    saber central block:
      saber block name: diffusion
      read:
        groups:
        - variables: [tocn, socn, ssh]
          horizontal:
            filepath: ./staticb/hz_ocean
          vertical:
            levels: {{MOM6_LEVS}}      # <------- BAD!!!!!
            filepath: ./staticb/vt_ocean

[guillaumevernieres]

Please fix the marine DA that you found out-

Sorry @apchoiCMD , we'll have to wait for upstream PR's to be merged before the marine DA is fixed.