WARNING: Set the "wind_thresh" and "wind_logic" configuration options to exclude zero vectors

[StevenSimon-NOAA]

When running EVS global_ens with the Met and MetPlus 6.0 Beta4, I encountered a new type of WARNING in all of the atmos g2o stats jobs.

WARNING: VL1L2Info::compute_stats() -> Skipping 1018 of 9243 vector pairs for which the direction difference is undefined.
WARNING: Set the "wind_thresh" and "wind_logic" configuration options to exclude zero vectors.

This seems to be tied to the VGRD_Z10_ENS_MEAN(,) versus VGRD/Z10 variables in today's global_ens g2o stats log files.

For what it's worth, this should not have an effect on the existing plotted global_ens stats, since we do not currently plot anything concerning wind direction.

I put the applicable log files in ftp://[email protected]:21/incoming/irap/met_help/simon_data/20240514_logs for your review.

[JohnHalleyGotway]

Hi @StevenSimon-NOAA. Thanks for speaking up about this warning message. We actually did notice this same behavior on another project last week and will plan to address it before the next development release of MET (MET-12.0.0-beta5).

Let me clarify what's happening.

The VL1L2 line type in MET contains vector partial sums computed from input U and V vector components. The VCNT line type contains statistics derived from those U and V vector components. As requested by NOAA/EMC, dtcenter/MET#2395 added new statistics to the VCNT line type for wind direction errors. It added 3 VL1L2 columns (DIR_ME, DIR_MAE, and DIR_MSE), 3 VAL1L2 columns (DIRA_ME, DIRA_MAE, and DIRA_MSE), and 12 new VCNT columns (DIR_ME, DIR_MAE, DIR_MSE, and DIR_RMSE plus confidence intervals).

Here's the problem. When U and V are both 0, wind direction is not defined. All of these line types include a TOTAL column to indicate the number of U and V matched pairs that went into them. Let's say you have 1000 U/V pairs for any given run, but 250 of them are 0-vectors for forecast, observation, or both. Reporting 1000 in the TOTAL column is no longer accurate. Yes, 1000 U/V pairs are used for the partial sums and wind speed statistics (for which 0-vectors have no impact), but only 750 of them were used for the wind direction statistics (for which 0-vectors are undefined). On other days the number of non-missing wind direction pairs could vary widely.

Fortunately, MET already provides a way to filter out 0-vectors using the wind_thresh and wind_logic options mentioned in that warning message. So setting wind_thresh = >0; for both forecast and observation and wind_logic = INTERSECTION; would discard any pairs where one or both of the wind directions are undefined.

But I realize this is not a very tenable situation. It certainly seems reasonable to look at all the partial sums and wind speed statistics without needing to discard the 0-vectors (which trigger that warning message) to do so.

So how would you like to proceed?

Of course, without any code changes, you could use those config options to side-step the problems that 0-vectors introduce.

And the easiest option would be to simply replace that warning message with a debug log message instead, so that MET still reports this issue without it setting off alarm bells for NCO.

A more involved, yet more mathematically correct, solution would be adding a new TOTAL_WDIR column to indicate the number of pairs containing non-zero vectors used to compute wind direction errors. Then we'd need to update Stat-Analysis to use those counts when aggregating results across multiple cases rather than the counts found in TOTAL. And this would have a trickle down impact on the logic in the METdataio and METcalcpy repositories.

Any recommendation on how NOAA/EMC would like us to address this issue?

[AliciaBentley-NOAA]

@JohnHalleyGotway @StevenSimon-NOAA I agree that the most mathematically correct way to address this issue would be to add a new TOTAL_WDIR column to indicate the number of pairs containing non-zero vectors used to compute wind direction errors and then update Stat-Analysis to use those counts when aggregating results across multiple cases rather than the counts found in TOTAL. That said, I don't know that that this work can/should be done for the beta5 release of MET-12.0.0/METplus-6.0.0. It has a lot of downstream impacts and it's a little far in the development process to address them all. Perhaps changing this message from WARNING to DEBUG would be helpful, since it satisfies the operational concerns about WARNING from NCO and also keeps the associated message in the log file.

I think that the best solution for EMC for right now would be to set wind_thresh = >0; for both forecast and observation and wind_logic = INTERSECTION; in order to discard any pairs where one or both of the wind directions are undefined. I don't know that we want "wind direction" stats when there is no wind. I'll share this link with Mallory, Yali, and Deanna (who also use wind direction in their verification) to see if this solution will work for them. Thanks!

[JohnHalleyGotway]

@AliciaBentley-NOAA I really appreciate the desire to not add to the list of work for the coordinated METplus-6.0.0 release. However, these new wind direction statistics were ADDED for the 6.0.0 release, so I'd very much prefer to "get them correct" for this release as well. Whenever possible, it's best to not split feature development across releases. That means that we'd have to update the database schema and aggregation logic not once (for 6.0.0), but twice (for 6.0.0 AND the next release).

As long as we're confident that adding the TOTAL_WDIR column and updating the corresponding aggregation logic is the right approach, I propose that we do so for 6.0.0. That'll make the output accurate and self-describing with no warning messages needed. And the fact that EMC doesn't need the 6.0.0 release until the end of the year gives us some extra time to get it correct for this release.

I'll just re-open the existing dtcenter/MET#2395 issue based on this discussion and plan to make these changes through a new PR for that issue. But I'll coordinate with the rest of the METplus team on the details.

Any objections?

[AliciaBentley-NOAA]

Thanks, John! I understand wanting to get it right the first time! I've CCed Yali, Mallory, and Deanna to see if they see any issues with your plan to update wind direction in the beta5 releases. I know that Mallory has already done some developing using the wind direction functionality using the beta4s, so she may have some thoughts on this. And Yali was the original wind direction requester, so she may have some thoughts on this as well. Deanna is out of town until next week, so we may not hear back from her before you decide to move forward on this. Thanks!

[ericgilleland]

@JohnHalleyGotway @StevenSimon-NOAA @AliciaBentley-NOAA I agree with Alicia that a new column should be added.

[StevenSimon-NOAA]

In the case where we would need to update our config files with wind_thresh and wind_logic, what would that look like in terms of syntax? It appears that the PointStat config files would be the applicable ones.

Would we simply be adding wind_thresh=>=0 and wind_logic=INTERSECTION to each PointStat config file? Or would this need to be thrown into an OVERRIDE option?

[AliciaBentley-NOAA]

@StevenSimon-NOAA If John and the DTC folks add what John would like to add, I don't think you would need to set those variables at all. Please see @JohnHalleyGotway 's latest comment above about how them plan to proceed, and let us know if that works for you.

[StevenSimon-NOAA]

If these WARNINGs are truly innocuous and have no bearing on the existing global_ens verification stats computations, I tend to agree that the path John and the other DTC folks laid out is the best approach going forward.

[malloryprow]

I think the TOTAL_WDIR is a good option. It preserves the meaning of TOTAL for the rest of the stats in the line, but it provides a clear picture for the wind direction.

Thanks for thinking up a solution @JohnHalleyGotway! Sorry for not thinking of this in the first development go around; the WMO stats requires thresholds and I hadn't considered the wider impact to the other EVS components that don't for wind. If you need anyone to test anything, let me know!

[JohnHalleyGotway]

FYI, I submitted a pull request (dtcenter/MET#2892) to address the issue described in this dtcenter/MET#2395 issue comment. I'll mark this discussion as being answered and recommend that we move any future discussion to comments on the dtcenter/MET#2395 issue.