New Use Case: construct use case verifying GFS cloud forecasts vs. GFS cloud analyses

[DanielAdriaansen]

Describe the New Use Case

This use case will demonstrate verifying forecasts of cloud information using the GFS global 0.25 degree grid using GridStat.

The fields to verify will be cloud fields we identify in GFS output such as:

• Cloud Fraction
• Cloud Base Height
• Cloud Top Height
• [potentially] Cloud Layers

We will need to search for and identify the proper field names and levels in the GFS files.

The measures of skill that should be included are:

• Gilbert Skill Score
• Equitable Threat Score
• Fraction Skill Score
• False Alarm Rate
• Hit Rate
• Bias

The end goal is for the user to be able to substitute the GFS analysis which we will use as "truth", with a separate GFS-based AI/ML cloud forecast product on the same GFS 0.25 degree grid. This framework is to support them to be able to do this. We may get some sample data of their truth, however due to restrictions on releasing the data we may need to leave the GFS analysis in place.

The user would also like to be able to stratify forecast performance based on categories of cloud types. These cloud types will be provided later on, but we should brainstorm another type of stratification we can perform using an external classification (maybe weather regimes? precipitation type?), or, implement some simple post-processing, for example stratify performance by all clouds e.g. >= 8 km ("high clouds").

Checklist to get working:

• Identify source of GFS global forecasts and analyses
• Identify available cloud variables
• Review existing cloud use cases for overlap and examples we may model from
• Begin configuration file development
• Begin documentation
• Begin testing GridStat w/METplus Wrappers
• Collaborate with New Use Case: construct use case verifying GFS cloud forecasts vs. GOES-16 cloud products #2744, New Use Case: construct use case verifying GFS cloud forecasts vs. ASOS ceiolometer #2745

Use Case Name and Category

`model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracBaseTop

Input Data

GFS 0.25 degree forecasts and analyses.
List input data types and sources.
Provide a total input file size, keeping necessary data to a minimum.

Acceptance Testing

Describe tests required for new functionality.
As use case develops, provide a run time here

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

Must be completed by 12/31/2024

Funding Source

7730022

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)
• Select privacy

Milestone and Projects

• Select Milestone as a METplus-Wrappers-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• For a METplus-Wrappers-X.Y.Z version, select the METplus-Wrappers-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

New Use Case Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• For any new datasets, an entry to the METplus Verification Datasets Guide.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: METplus-Wrappers-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept your changes. Merge branch into develop.
• Create a second pull request to merge develop into develop-ref, following the same steps for the first pull request.
• Delete your fork or branch.
• Close this issue.

[DanielAdriaansen]

GFS data are here:
/d1/projects/METplus/METplus_Data/development/nrl/cloud/GFS_0.25

Using this command:

wgrib2 -v gfs.0p25.2024030700.f012.grib2 | grep cloud

I see relevant variables:

630:450488137:d=2024030700:LCDC Low Cloud Cover [%]:low cloud layer:12 hour fcst:
631:451333325:d=2024030700:LCDC Low Cloud Cover [%]:low cloud layer:6-12 hour ave fcst:
632:452258290:d=2024030700:MCDC Medium Cloud Cover [%]:middle cloud layer:12 hour fcst:
633:452866413:d=2024030700:MCDC Medium Cloud Cover [%]:middle cloud layer:6-12 hour ave fcst:
634:453577593:d=2024030700:HCDC High Cloud Cover [%]:high cloud layer:12 hour fcst:
635:454309341:d=2024030700:HCDC High Cloud Cover [%]:high cloud layer:6-12 hour ave fcst:
638:457005258:d=2024030700:HGT Geopotential Height [gpm]:cloud ceiling:12 hour fcst:
639:458203404:d=2024030700:PRES Pressure [Pa]:convective cloud bottom level:12 hour fcst:
640:458750807:d=2024030700:PRES Pressure [Pa]:low cloud bottom level:6-12 hour ave fcst:
641:460188975:d=2024030700:PRES Pressure [Pa]:middle cloud bottom level:6-12 hour ave fcst:
642:461414641:d=2024030700:PRES Pressure [Pa]:high cloud bottom level:6-12 hour ave fcst:
643:462935144:d=2024030700:PRES Pressure [Pa]:convective cloud top level:12 hour fcst:
644:463536141:d=2024030700:PRES Pressure [Pa]:low cloud top level:6-12 hour ave fcst:
645:464992169:d=2024030700:PRES Pressure [Pa]:middle cloud top level:6-12 hour ave fcst:
646:466180239:d=2024030700:PRES Pressure [Pa]:high cloud top level:6-12 hour ave fcst:
647:467707298:d=2024030700:TMP Temperature [K]:low cloud top level:6-12 hour ave fcst:
648:468725963:d=2024030700:TMP Temperature [K]:middle cloud top level:6-12 hour ave fcst:
649:469628288:d=2024030700:TMP Temperature [K]:high cloud top level:6-12 hour ave fcst:
650:470836489:d=2024030700:TCDC Total Cloud Cover [%]:convective cloud layer:12 hour fcst:
651:471552157:d=2024030700:TCDC Total Cloud Cover [%]:boundary layer cloud layer:6-12 hour ave fcst:

In addition, there is an isobaric "total cloud cover" variables (TCDC) at various pressure levels in the file.

[j-opatz]

@CPKalb you and I may need to tackle this one; A GFS-to-GFS gridded comparison seems...too basic. We have plenty of examples where this type of verification is completed, including some of the cloud verification use cases (some of which are more complex and still adaptable).

I think in the last NRL meeting we discussed looping in a MODE analysis/DMAP approach on this; I'd like for both of us to do some research and see if there's any cloud-identification research that could shed a light on a path forward. I'll try to find some time to meet up on Fri-early Mon to discuss our findings and settle on an approach.

[CPKalb]

We have a use case that uses distance maps on brightness temperatures. I can't remember if there was a paper or anything associated with this though. I'll have to poke around.
https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/GridStat_fcstFV3_obsGOES_BrightnessTempDmap.html#sphx-glr-generated-model-applications-short-range-gridstat-fcstfv3-obsgoes-brightnesstempdmap-py

https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/MODE_fcstFV3_obsGOES_BrightnessTempObjs.html#sphx-glr-generated-model-applications-short-range-mode-fcstfv3-obsgoes-brightnesstempobjs-py

[j-opatz]

Adding notes here so direction of use case is recorded:

It sounds like (for the time being) we should just focus on a cloud fraction comparison between the two and exhibit the difference between traditional statistics and an object based approach. So the use case would be a GridStat call and a MODE call, looking at 1 (or more) of the fractional cloud coverage heights (low, mid, high, or convective).

[CPKalb]

Is there a date I should start with for this verification or should I just pick something?

[DanielAdriaansen]

Is there a date I should start with for this verification or should I just pick something?

Dates are discussed here: https://docs.google.com/document/d/1SJb2ZBrLG9jqH_Tu_ssB8lqeDxnPx1Xp-FHFRdj7HM0/edit?tab=t.0#heading=h.76lq03tjjtx6

3/7/2024 00 UTC initialization with forecasts leads of 6,12,18h and valid times of 06, 12, and 18 UTC on 3/7/2024.