Add 187/287 METAR config templates

[spanNOAA]

Finalize yamls for 187/287 metar observations with reasonable results.

See issue #46 for details.

[SamuelDegelia-NOAA]

Thanks @spanNOAA! One thing I noticed is that the filter order is slightly different from the mesonet yaml files (e.g., here). The mesonet obs have temporal thinning (duplicate check) at the end while these yamls have it towards the beginning. Do you know if that order matters?

Also, could we add the online domain check here? That filter is commented out of the q/uv yamls but we should add it to all of them.

[delippi]

Thanks @spanNOAA! One thing I noticed is that the filter order is slightly different from the mesonet yaml files (e.g., here). The mesonet obs have temporal thinning (duplicate check) at the end while these yamls have it towards the beginning. Do you know if that order matters?

Also, could we add the online domain check here? That filter is commented out of the q/uv yamls but we should add it to all of them.

I think the order does matter. I'm not 100% sure what the correct order should be. It seems to me that GSI has the duplicate checks earlier in the code, so I did try putting the duplicate filter toward the top of the yaml. The problem I was seeing there is you could be throwing out obs via the duplicate filter that are better quality and the lesser quality fail the gross checks. If it is near the bottom, then you allow all obs to go through the gross checks first and then pick from those which duplicates to keep. While writing this, I'm starting to think maybe we do want the more restrictive GSI method?

[spanNOAA]

Thanks @SamuelDegelia-NOAA @delippi pointing that out! I hadn't noticed this difference before, probably because of the limited number of METAR observations, but it's definitely a good catch regarding the order issue. I’ve changed the order, and the result hasn't changed—the cost function at the last step is exactly the same. I'll update the order to match yours for consistency.
I still have some questions about the online domain check. Is it meant to replace the offline domain check? How should I properly set the min/max values for the observable_domain_mask? Do you have any instructions on how to use this function?

[delippi]

When I updated the submodules in PR #227, that should have fixed issues we were having with the online domain check which should now be able to be used alone. There should no longer be a need to use the offline domain check... Although it can still be helpful for debugging.

This is how you use the online domain check according to @HuiLiu-NOAA. I'm not sure exactly what the min/max values are doing.

         # Online domain check
         - filter: Bounds Check
           filter variables:
           - name: airTemperature
           test variables:
           - name:  GeoVaLs/observable_domain_mask
           minvalue: 0.0
           maxvalue: 0.5

I've placed this filter as one of the first filters and right after the "Time window filter" (the one that uses timeOffset). The main thing is that it just needs to be used before ObsErrorFactorPressureCheck.

[spanNOAA]

@HuiLiu-NOAA Could you provide more information about the observable_domain_mask? Specifically, what do the minimum and maximum values represent? Thanks.

[HuiLiu-NOAA]

@HuiLiu-NOAA Could you provide more information about the observable_domain_mask? Specifically, what do the minimum and maximum values represent? Thanks.

Yes. Here are the details:
For MPAS, the domain mask has values of 0-7; where 0=interior, 1-4=relax zone points, and 5-7=prescribed zone. Observations in these different zones could be used/evaluated in DA. Current, the interpolated value of the mask at OBS location with max=0.5 include only the OBS inside domain interior for DA. The max value can be relaxed to any values between 1.0 and 7.0 to consider/include OBS in the appropriate zones for DA.

For FV3, its domain mask is relative simple right now, 0=interior, 1=outside of domain. So, the interpolated mask value at OBS location with max=0.5 consider only the OBS inside the FV3 domain. This could be changed in future development.

[spanNOAA]

@HuiLiu-NOAA Thanks for the details. @SamuelDegelia-NOAA When applying the online check, we do have more data points at the edge of the domain compared to the offline check.

[SamuelDegelia-NOAA]

@HuiLiu-NOAA Thanks for the details. @SamuelDegelia-NOAA When applying the online check, we do have more data points at the edge of the domain compared to the offline check.

That's great! Even if its a small difference, glad that we don't have to remove any obs inside the domain.

[delippi]

@HuiLiu-NOAA Could you provide more information about the observable_domain_mask? Specifically, what do the minimum and maximum values represent? Thanks.

Yes. Here are the details: For MPAS, the domain mask has values of 0-7; where 0=interior, 1-4=relax zone points, and 5-7=prescribed zone. Observations in these different zones could be used/evaluated in DA. Current, the interpolated value of the mask at OBS location with max=0.5 include only the OBS inside domain interior for DA. The max value can be relaxed to any values between 1.0 and 7.0 to consider/include OBS in the appropriate zones for DA.

For FV3, its domain mask is relative simple right now, 0=interior, 1=outside of domain. So, the interpolated mask value at OBS location with max=0.5 consider only the OBS inside the FV3 domain. This could be changed in future development.

@HuiLiu-NOAA, thanks for this info. Does this update the QCflagData group after it is run? My understanding is that QCflagData are up-to-date values (set by one of the preceding filters). EffectiveQC are final values obtained after execution of all filters; this group does not exist while filters are running, but is present in the output ioda file.

[HuiLiu-NOAA]

@HuiLiu-NOAA Could you provide more information about the observable_domain_mask? Specifically, what do the minimum and maximum values represent? Thanks.

Yes. Here are the details: For MPAS, the domain mask has values of 0-7; where 0=interior, 1-4=relax zone points, and 5-7=prescribed zone. Observations in these different zones could be used/evaluated in DA. Current, the interpolated value of the mask at OBS location with max=0.5 include only the OBS inside domain interior for DA. The max value can be relaxed to any values between 1.0 and 7.0 to consider/include OBS in the appropriate zones for DA.
For FV3, its domain mask is relative simple right now, 0=interior, 1=outside of domain. So, the interpolated mask value at OBS location with max=0.5 consider only the OBS inside the FV3 domain. This could be changed in future development.

@HuiLiu-NOAA, thanks for this info. Does this update the QCflagData group after it is run? My understanding is that QCflagData are up-to-date values (set by one of the preceding filters). EffectiveQC are final values obtained after execution of all filters; this group does not exist while filters are running, but is present in the output ioda file.

I believe this updates the QCflagData group after it is run. This can be verified by check the QCflags in the diag file.

[ShunLiu-NOAA]

@delippi could you please review the updates of this PR?

[delippi]

Hi @spanNOAA, I'm so sorry for all the picky comments. There were many comments that I made in the airTemperature yaml that also apply to the other yamls but I did not add those comments to the other yamls; I was trying to keep picky, repetitive comments to a minimum. So when you make a change for the airTemperature yaml, can you also check to see if it applies to the others? You can also use vi -d msonet_stationPressure_188.yaml adpsfc_stationPressure_187.yaml (for example) to compare. It would be nice if differences were kept as few as possible! This makes debugging and maintenance of these yamls much much easier!

Overall, everything looks very nice! Thanks for putting these together and mostly following the mesonet yamls. I think the differences here were just based on older versions of the mesonet yamls which recent updates never got propagated into these yamls.

[delippi]

@spanNOAA, thanks for making all those changes!