Internal: Determine/document a process to test use cases that can't run in automated test environment

[georgemccabe]

Currently there are a few use case that are not run in GitHub Actions for various reasons, like they exceed the memory limit or disk space limit. Some of these are noted in the Contributor's Guide. As part of the release process, we should make sure to run these use cases outside of the GitHub Actions automated testing environment to ensure that they run as expected.

The Contributor's Guide describes the process to add use cases that can't be run by not assigning a number to the use case in the internal/tests/use_cases/all_use_cases.txt file and excluding them from the .github/parm/use_case_groups.json file that determines the groups of use cases to run in the automated tests. We have since added support for a "disabled" key in the use_case_groups.json file to always skip those use cases.

There are a few use cases that are not run in the automated tests, but the reason is unclear because they are not listed in the Contributor's Guide section. This could happen for various external reasons (unrelated to GitHub Actions limitations) and we may be planning on eventually getting these cases working again.

Describe the Task

• Update the documentation and process for adding use cases that can't be run in GHA to:
  • Assign a number
  • Mark them as disabled in the JSON use case groups file
  • Add a reason why it is disabled for internal reference so we know if it will never be able to run in GHA or not
• Develop a process to run the use cases that can be run outside of GHA and run them to test that they work (on seneca?)
• Document process to run the cases and see if this step can be scripted to make this testing easier for future releases

Time Estimate

This depends on how much of this work we want to complete for this release and which should move to another issue for future work. We should at very least test the use cases that need to be tested!

Sub-Issues

Consider breaking the task down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects

• Select Milestone as a METplus-Wrappers-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• For a METplus-Wrappers-X.Y.Z version, select the METplus-Wrappers-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Task Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• For any new datasets, an entry to the METplus Verification Datasets Guide.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: METplus-Wrappers-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[georgemccabe]

For now, I will start with testing the use cases that are not run in GHA on seneca. Some of the use cases have input data that is provided with the rest of the data for the model_applications categories. Some of them have data in an additional tar file on the web.

• marine_and_cryosphere:11 GridStat_fcstRTOFS_obsGHRSST_climWOA_sst

  • Ran without errors
  • Ran on seneca using command:

    run_metplus.py \
    ~/METplus/parm/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.conf \
    ~/mccabe.seneca.conf \
    user_env_vars.MET_PYTHON_EXE=/home/met_test/miniforge3/envs/icecover.v6.0/bin/python3
    

• s2s_stratosphere:1 UserScript_fcstGFS_obsERA_StratospherePolar

  • Data found in https://dtcenter.ucar.edu/dfiles/code/METplus/METplus_Data/v6.0/additional_data_UserScript_fcstGFS_obsERA_StratospherePolar.tgz
  • Added data to seneca:/d1/projects/METplus/METplus_Data/not_in_gha
  • Ran without errors (after bugfix to METcalcpy)
  • Ran on seneca using command:

    mamba activate weatherregime.v6.0
    run_metplus.py \
    ~/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf \
    ~/mccabe.seneca.conf \
    config.INPUT_BASE=/d1/projects/METplus/METplus_Data/not_in_gha \
    config.OUTPUT_BASE=/d1/personal/mccabe/out_s2s_strat \
    user_env_vars.PYTHONPATH=/home/mccabe/METcalcpy:/home/mccabe/METdataio:/home/mccabe/METplotpy
    

• s2s_stratosphere:2 UserScript_fcstGFS_obsERA_StratosphereQBO

  • Data found in https://dtcenter.ucar.edu/dfiles/code/METplus/METplus_Data/v6.0/additional_data_UserScript_fcstGFS_obsERA_StratosphereQBO.tgz
  • Added data to seneca:/d1/projects/METplus/METplus_Data/not_in_gha
  • Ran without errors
  • Ran on seneca using command:

    mamba activate weatherregime.v6.0
    run_metplus.py \
    ~/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereQBO.conf \
    ~/mccabe.seneca.conf \
    config.INPUT_BASE=/d1/projects/METplus/METplus_Data/not_in_gha \
    config.OUTPUT_BASE=/d1/personal/mccabe/out_s2s_strat2 \
    user_env_vars.PYTHONPATH=/home/mccabe/METcalcpy:/home/mccabe/METdataio:/home/mccabe/METplotpy
    

• short_range:14 UserScript_fcstRRFS_fcstOnly_Reformat_Aggregate_Plot

  • Disabled reason: "need new RRFS data with new line types"
  • Not expected to successfully run because we need new input data
  • Ran without errors (after a few minor changes to use case scripts)
  • Ran on seneca using command:

    mamba activate rrfs_usecase_env
    run_metplus.py \
    ~/METplus/parm/use_cases/model_applications/short_range/UserScript_fcstRRFS_fcstOnly_Reformat_Aggregate_Plot.conf \
    ~/mccabe.seneca.conf \
    config.OUTPUT_BASE=/d1/personal/mccabe/out_short_range14 \
    user_env_vars.PYTHONPATH=/home/mccabe/METcalcpy:/home/mccabe/METdataio:/home/mccabe/METplotpy
    

  • Note: will need to create a conda environment in internal/scripts/docker_env for this use case!

• short_range:15 MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning

  • Ran without errors
  • Ran on seneca using command:

    run_metplus.py \
    ~/METplus/parm/use_cases/model_applications/short_range/MODEMultivar_fcstRRFS_obsGOES_MRMS_BrightnessTemp_Lightning.conf \
    ~/mccabe.seneca.conf
    

I ran each of these using today's MET nightly build directory on seneca. I automate this by setting an environment variable in my .bashrc:
export TODAY_YMD=$(date +%Y%m%d)
then set the following in mccabe.seneca.conf:
MET_INSTALL_DIR=/d1/projects/MET/MET_regression/develop/NB{ENV[TODAY_YMD]}/MET-develop