Declaring the Raw NWM Data Source

Table of Contents

• What evaluation declaration is required to read raw NWM data?
• Are there example declarations available?
  • Example declaration for NWM short-range forecasts
  • Example declaration for NWM single-valued, medium-range forecasts
  • Example declaration for NWM ensemble, medium-range forecasts using the legacy NWM Medium Range Ensemble
  • Example for another variable, such as qSfcLatRunoff:
  • Example combining the NWM medium-range ensemble forecasts for NWM v2.1 and v2.2 into a single evaluation (since the model was unchanged between those versions):
  • Example use of nomads
  • Example use of Amazon S3

Raw NWM data is available in a netCDF format, with the files organized in a specific directory structure and file names following a specific convention.

What evaluation declaration is required to read raw NWM data?

In all cases, whether using the WRDS-Store archive at the National Water Center, a filesystem, s3 bucket (e.g., AWS), or Nomads, a reference_dates range is required in the declaration so that the WRES software can set the scope of requests to open and read datasets.

The interface must also be specified. Interface options starting with nwm are short-hand for exact netCDF blob layouts with particular attributes. While extensive testing has been done of the “streamflow” variable, other variables can be declared as long as they were packed into the netCDF data the same way as the streamflow data. To get a list of variables available in a particular dataset, download a sample netCDF and run the tool ncdump to obtain the header. For example, visit https://nomads.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/ and go to one of the available dates and datasets, download a netCDF, use ncdump -h [netCDF file] to see variables. Note that the netCDF reader in WRES assumes a 32-bit integer value packed by either a 32-bit float or 64-bit double scale and offset value. The enumeration of available interface strings are found in the YAML (as of WRES Release 6.14) schema:

https://github.com/NOAA-OWP/wres/blob/master/wres-config/nonsrc/schema.yml (search for SourceInterfaceEnum).

The variable for the sources must be declared, as it defines verbatim the data variable to be obtained from the netCDF.

The path is a prefix up until the point where the netCDF blob layout changes, which is the nwm.yyyyMMdd directory/folder/bucket. The date ranges from the issued datetimes combined with the interface short-hand is used to fill out the remainder of the paths for all blobs. The feature_id variable must exist in the netCDF blobs. The variable specified by the user in the declaration variable tag will be requested verbatim, so this variable must exist in the netCDF. The units attribute must exist for the variable in the netCDF. The features WRES requests from the dataset will be found from the WRES project declaration’s features tag, either directly from an NWM feature id declared or indirectly from the WRES feature correlations that WRES is aware of.

Short-range, medium-range deterministic, medium-range ensemble, long-range, and analysis-and-assimilation can be read by the WRES. The analyses are less straightforward to configure.

Are there example declarations available?

Below are examples of declaring the use of raw NWM data in a netCDF format.

All of the examples below are written assuming NWM data is to be obtained from an on-line source. To read from a local disk, change "https://" to "file://" and modify the path that follows accordingly. Furthermore, NWM 3.0 data is being read, in most cases, with the directory structure assumed below that point. Lastly, only the CONUS data is evaluated in the examples that follow. However, interface options are also available for Puerto Rico, Hawaii, and Alaska. Please review the schema at the link provided above for a complete listing.

Example declaration for NWM short-range forecasts

    predicted:
      label: NWM Short Range
      sources:
        - uri: https://[omitted URL]/nwm/3.0
          interface: nwm short range channel rt conus
      variable: streamflow

Example declaration for NWM single-valued, medium-range forecasts

predicted:
  label: NWM Medium Range Deterministic
  sources:
    - uri: https://[omitted URL]/nwm/3.0
      interface: nwm medium range deterministic channel rt conus hourly
  variable: streamflow

NOTE: To use the NWM Medium Range Deterministic v2.1 and later hourly data, use interface nwm medium range deterministic channel rt conus hourly. Use of nwm medium range deterministic channel rt conus, designed for NWM v1.2 and earlier data, will result in every third hour being evaluated (NWM forecasts were for every three hours with NWM v1.2 and earlier).

Example declaration for NWM ensemble, medium-range forecasts using the legacy NWM Medium Range Ensemble

predicted:
  label: NWM Medium Range Ensemble
  sources:
    - uri: https://[omitted URL]/nwm/3.0
      interface: nwm medium range ensemble channel rt conus hourly
  variable: streamflow

NOTE: To use the NWM Medium Range Deterministic v2.1 and later hourly data, use interface nwm medium range ensemble channel rt conus hourly. Use of nwm medium range ensemble channel rt conus, designed for NWM v1.2 and earlier data, will result in every third hour being evaluated (NWM forecasts were for every three hours with NWM v2.0 and earlier).

Example for another variable, such as qSfcLatRunoff:

predicted:
  label: NWM Short Range
  sources:
    - uri: https://[omitted URL]/nwm/3.0
      interface: nwm short range channel rt conus
  variable: qSfcLatRunoff

Example combining the NWM medium-range ensemble forecasts for NWM v2.1 and v2.2 into a single evaluation (since the model was unchanged between those versions):

predicted:
  label: NWM Medium Range Ensemble
  sources:
    - uri: https://[omitted URL]/nwm/2.2
      interface: nwm medium range deterministic channel rt conus hourly
    - uri: https://[omitted URL]/nwm/2.1
      interface: nwm medium range deterministic channel rt conus hourly
  variable: streamflow

Example use of nomads

Nomads has one or two days available, constantly changing date-range availability:

predicted:
  label: NWM Long Range Ensemble
  sources:
    - uri: https://nomads.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/
      interface: nwm long range channel rt conus
  variable: streamflow

Example use of Amazon S3

Amazon S3’s noaa-nwm-pds bucket has around 30 days available, constantly changing availability:

predicted:
  label: NWM Short Range
  sources:
    - uri: https://s3.amazonaws.com/noaa-nwm-pds/
      interface: nwm short range channel rt conus
  variable: streamflow