Ingest and dump MRMS radar data for RTMA model

[ilianagenkova]

MRMS radar data was requested by RTMA model team.
Work plan
A) Gather info on data (provider, format, contents)
B) Gather info on user/RTMA team needs (variables, formatting, frequency, etc)
C) Leverage off radarl2 package - how is data pre-processed, what can we learn from it
D) Leverage off existing satingest routines to write MRMS ingests code and generate tanks
E) Expand obsproc to dump MRMS data tanks

@PraveenKumar-NOAA, If you copy/paste the info we gathered already (from emails), you will have one-stop document to refer to.

[PraveenKumar-NOAA]

The web link for the MRMS, radarl2, and staingest packages:

• https://mrms.nssl.noaa.gov
• https://github.com/NOAA-EMC/radarl2
• https://github.com/NOAA-EMC/satingest

A couple of important information about the radarl2 package and processing:

• The source of the radar data:
  ${DCOMROOT}/ldmdata/obs/upperair/nexrad_level2

• The format of the raw incoming radar data:
  binary format raw data with bz2 compressed

• The tank used to run Obsproc:
  radar_level2.$radar_level2_ver is for dumping data to tank b006

• User of this data:
  NAM, RAP, HRRR, GDAS, and verification package

[PraveenKumar-NOAA]

MRMS/radarl2 variables information:

There is MRMS data available on WCOSS2 in the following directory: /lfs/h1/ops/prod/dcom/ldmdata/obs/upperair/mrms, in grib2 format.

A few netCDF files (pe0076.conv_uv_03.nc4 and Gridded_ref.nc) were used from the following directory, /lfs/h2/emc/stmp/Shun.Liu/stmp/tmpnwprd/rrfs_a/2022101800, to get information about the variables:

[ilianagenkova]

@PraveenKumar-NOAA,
ls -l /lfs/h1/ops/prod/dcom/ldmdata/obs/upperair/mrms/conus
total 20412
drwxr-sr-x 2 dfprod prod 749568 Oct 19 03:56 EchoTop
drwxr-sr-x 2 dfprod prod 446464 Oct 19 03:56 LowLevelCompositeReflectivity
drwxr-sr-x 2 dfprod prod 13905920 Oct 19 03:56 MergedReflectivityQC
drwxr-sr-x 2 dfprod prod 471040 Oct 19 03:55 MergedReflectivityQCComposite
drwxr-sr-x 2 dfprod prod 495616 Oct 19 03:57 MergedReflectivityQComposite
drwxr-sr-x 2 dfprod prod 487424 Oct 19 03:55 MESH
drwxr-sr-x 2 dfprod prod 253952 Oct 19 03:17 MultiSensorQPE
drwxr-sr-x 2 dfprod prod 331776 Oct 19 03:55 POSH
drwxr-sr-x 2 dfprod prod 376832 Oct 19 03:55 PrecipFlag
drwxr-sr-x 2 dfprod prod 380928 Oct 19 03:56 PrecipRate
drwxr-sr-x 2 dfprod prod 413696 Oct 19 03:56 RadarOnly_QPE
drwxr-sr-x 2 dfprod prod 409600 Oct 19 03:55 RadarQualityIndex
drwxr-sr-x 2 dfprod prod 1060864 Oct 19 03:55 RotationTrack
drwxr-sr-x 2 dfprod prod 364544 Oct 19 03:57 SeamlessHSR
drwxr-sr-x 2 dfprod prod 344064 Oct 19 03:55 SHI
drwxr-sr-x 2 dfprod prod 327680 Oct 19 03:55 VIL

These are the variables available from MRMS. If you list each dir, you'll see these files come every few minutes, like Shun told us.

[PraveenKumar-NOAA]

MRMS data directory structures on WCOSS2:

There are five mrms sub-directories named as:

• alaska
• carib
• conus
• guam
• hawaii

List of variables from mrms/conus grib2 files:

• conus sub-directory is used to get the variable information.

• There are a total of EIGHTEEN variables, including LAT and LON, which are shown below in the following format, (Name, Long Name, Type) using Panoply's output.

[ilianagenkova]

@PraveenKumar-NOAA , it is not clear to me what the "pe0076.conv_uv_03.nc4 and Gridded_ref.nc" files are, i.e. how were they generated - are they output from Shun's code? are they the netcdf version of the grib2-format files on wcoss2 ?

[JacobCarley-NOAA]

I am pretty sure that pe0076.conv_uv_03.nc4 is output from GSI. Likewise, Gridded_ref.nc is output from a pre-processing step for GSI that interpolates (and optionally thins) the MRMS gridded radar reflectivity to the model grid, in preparation for assimilation.

[PraveenKumar-NOAA]

@ilianagenkova these files are output from Shun's code taken from the following directory: /lfs/h2/emc/stmp/Shun.Liu/stmp/tmpnwprd/rrfs_a/2022101800), and I don't know how he generated these. But these files are not a direct product of the MRMS data.

@JacobCarley-NOAA thank you for the clarification.

[ilianagenkova]

But we don't care about Shun's code, do we? I thought he generates nexrad dumps with lower res data, and they have only a couple of variables. The reason we have the MRMS ingest task is to give the users high-res radar data and more parameters (atmos state variables). So again, we need to know what's in the: $DCOMROOT/ldmdata/obs/upperair/mrms/conus/* files. Go through the list of files, for example - $DCOMROOT/ldmdata/obs/upperair/mrms/conus/MergedReflectivityQCComposite/MergedReflectivityQCComposite_00.50_20221122-165238.grib2.gz and pull the variables from the grib2 files. Some variables we will simply ingest, while others might need to be computed, is my understanding.

…

On Tue, Nov 22, 2022 at 10:15 AM PraveenKumar-NOAA ***@***.***> wrote: @ilianagenkova <https://github.com/ilianagenkova> these files are output from Shun's code taken from the following directory: /lfs/h2/emc/stmp/Shun.Liu/stmp/tmpnwprd/rrfs_a/2022101800), and I don't know how he generated these. But these files are not a direct product of the MRMS data. @JacobCarley-NOAA <https://github.com/JacobCarley-NOAA> thank you for the clarification. — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOC4YXTDN5OI5PO73RGHTULWJTWTNANCNFSM6AAAAAAQ5AZ4FQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JacobCarley-NOAA]

@ilianagenkova We might be conflating a couple different things. pe0076.conv_uv_03.nc4 is diagnostic output from the GSI. Probably not of interest. And Gridded_ref.nc uses MRMS input directly. It is not related to any of the nexrad level2 radial wind dumps and is likely out of scope for this project.

[PraveenKumar-NOAA]

@ilianagenkova @JacobCarley-NOAA I extracted variables information from the MRMS data, which are listed in the following document, https://docs.google.com/document/d/1Uud8JMDnMLgk_kkMPZCJm1HaAchYurjDl1W38YBk8Bg/edit?pli=1. There are four main variables, latitude, longitude, time, and altitude_above_msl, and I think the rest are the computed ones.

@ilianagenkova I listed variables information from Shun's code, i.e. radarl2 package at the very beginning., at that time I was not sure how to proceed. No, we don't care about Shun's code as far as my understanding.

[ilianagenkova]

Next steps:

• will NSSL run the radial wind decoder and provide the winds in grib2 format (like what we get in /ldmdata) ? or we need to make the decoder part of satingest (more work!)
• can RRFS and RTMA users read MRMS data from grib2, netcdf, bufr?

[PraveenKumar-NOAA]

• A ticket (#RITM0236221) is opened for Dataflow to get the Radial Wind data (probably in NetCDF).
• Both RRFS and RTMA users can read MRMS data from the NetCDF file.

Once Radial Wind data is available, the next step will be:

• Ingest the MRMS data for Reflectivity, MergedReflectivityQComposite, and Radial Wind. The new functionality will be added to the satingest codes for this process.

[PraveenKumar-NOAA]

Requested sample radial wind data from MRMS folks.

[PraveenKumar-NOAA]

This issue is on hold until the radial wind data is provided on WCOSS2 by the Dataflow team or radial wind data samples are provided by the MRMS folks.

[PraveenKumar-NOAA]

@JacobCarley-NOAA @ilianagenkova most of the data required by RTMA users is already available on WCOSS2. If no further action is needed, this issue can be closed. Thank you.

[JacobCarley-NOAA]

@PraveenKumar-NOAA I missed the tag in GitHub - apologies.

The reflectivity files are available - but are the winds also available?

[PraveenKumar-NOAA]

@JacobCarley-NOAA no, a Dataflow ticket is opened for the wind. A request is also made for the sample wind data.

[JacobCarley-NOAA]

Thanks Praveen - makes sense.

Was the request for the sample wind data made directly to the MRMS group? Am wondering if we ought to follow up if it's been a little while.

[PraveenKumar-NOAA]

Yes, it has been made directly to the MRMS group. I will send a reminder again.

[ilianagenkova]

@JacobCarley-NOAA , I had advised @PraveenKumar-NOAA to make inquiries every two weeks at first, then we cut them down to monthly...I believe Anthony is very aware of our interest in the radial wind by now :)

[JacobCarley-NOAA]

@ilianagenkova Ha! Hopefully he hasn't set up a spam filter for us :-)

[PraveenKumar-NOAA]

@ilianagenkova finally with the help of Jacob and Jian, I got the sample radial wind data, which is located into the following folder, EMC_radial_velocity.

[PraveenKumar-NOAA]

The sample radial velocity (Vr) data (both "raw" and dealiased) is from KCXX radar. The data are in radar coordinates and in NetCDF format. There are also example images of the raw ("Aliased Velocity") and dealiased ("Velocity") fields in this folder for the reference.

Additional information about the data:

1. the MRMS dealiased velocity does not include any QC or masking of non-hydrometeor echoes (including clear air, birds and insects). It's just a dealiased version of the raw Vr field.
2. the NEXRAD level-3 product suite also provides dealiased velocities (no QC or masking) up to 3.4 deg elevation angle. The level-3 dealiasing scheme is slightly newer than the MRMS version and obtaining the level-3 data in real-time may be faster than getting the MRMS single radar Vr fields. The drawback with the level-3 is the lack of the higher tilts.
3. NSSL is developing a so-called "point cloud" data set that combines 3D Vr fields from all the radars (e.g., in CONUS). This may take a while to implement in the operations.

[PraveenKumar-NOAA]

This issue is on hold until we get further input from the MRMS folks.

[PraveenKumar-NOAA]

Sample data file were received in NetCDF format. It's a very small set of single radar (dealiased) Velocity data.

Details:
This is a grid algorithm that then samples the current radar volume to generate data values per grid cell.

The idea is to eventually concatenate multiple radar's output into a single file if possible, so that each grid cell might have multiple values relating to a contributing radar.

It uses Cressman sampling around the 'hit' gate (8 azimuth, 12 gate) instead of averaging values. The size and method of sampling will eventually be configurable assuming this technique gives us good results.

Eventually the final concatenated file would include a radar list and you would have an ID array telling you the radar for each sample, something like this:
RadarList: KTLX, KOUN
Latitude: Lat1, Lat2, ...
Longitude: Lon, Lon2, ...
RadarID: 0, 1, 0, 0
Or you could do KTLXLatitude, KOUNLatitude something like that. Open to ideas on the best approach.

We are using a defined 3D grid in space of Lat, Lon and Height which can be defined by us to the desired resolution.
The x for example is the arc length from the 3D cell point to the radar center:
const float x = fabs(oLonRad - rLonRad) * earthR; // Arc length in meters
const float y = fabs(oLatRad - rLatRad) * earthR; // Arc length in meters
const float z = fabs(oHtKMs - rHtKMs) * 1000.0; // Meters
const float range = sqrtf(x * x + y * y + z * z); // Distance formula
const float ux = x / range; // units 'should' cancel so technically we're dimensionless
const float uy = y / range;
const float uz = z / range;

The velocity directions are all based on the contributing radar center. 'Mapped' refers to the defined 3D grid, we get a velocity 'sample' bases on the location in the 3D grid in relation to the nearest tilt in the volume.

ALTERNATIVE METHODS:
We have another technique which is based on the 'pointcloud' idea where we do it in pure polar. The downside to the polar method is losing some functionality such as terrain ability, nearest N radars, etc. in the data. But we can experiment with different ideas.

Next Step: The data files are shared with the RRFS/[3D]-RTMA uses to check the quality of the data. If data quality is satisfied, a DataFlow ticket will be submitted!

[PraveenKumar-NOAA]

The variable information extracted from one of the NetCDF file is show below:

netcdf \20240807-190020.000 {
dimensions:
Values = 58450 ;
variables:
float MappedVelocity(Values) ;
MappedVelocity:units = "MetersPerSecond" ;
MappedVelocity:Units = "MetersPerSecond" ;
float ux(Values) ;
ux:units = "dimensionless" ;
ux:Units = "dimensionless" ;
float uy(Values) ;
uy:units = "dimensionless" ;
uy:Units = "dimensionless" ;
float uz(Values) ;
uz:units = "dimensionless" ;
uz:Units = "dimensionless" ;
float latitude(Values) ;
latitude:units = "Degrees" ;
latitude:Units = "Degrees" ;
float longitude(Values) ;
longitude:units = "Degrees" ;
longitude:Units = "Degrees" ;
float height(Values) ;
height:units = "Meters" ;
height:Units = "Meters" ;

// global attributes:
:MRMSWriterInfo = "RAPIO (Build: 2024-08-07)" ;
:Unit-unit = "dimensionless" ;
:Unit-value = "MetersPerSecond" ;
:Sourcename = "KMHX" ;
:SubType-unit = "dimensionless" ;
:SubType-value = "S2" ;
:TypeName = "MappedVelocity" ;
:DataType = "PointTable" ;
:Latitude = 34.7757987976074 ;
:Longitude = -76.8764038085938 ;
:Height = 43.9999997615814 ;
:Time = 1723057220 ;
:FractionalTime = 0. ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
:attributes = " Unit SubType" ;

[ilianagenkova]

@PraveenKumar-NOAA , thanks for the update on the limited data samples, located at EMC_radial_velocity.

I am adding @ShunLiu-NOAA and @ManuelPondeca-NOAA as "assignees" to get their attention and input.

@JacobCarley-NOAA , shall we remove you from this ticket?

[PraveenKumar-NOAA]

@ilianagenkova Thanks! I already shared the data with @ManuelPondeca-NOAA and @ShunLiu-NOAA to get their input on its quality. @ShunLiu-NOAA and his team are already working on it.