Correct time_summary settings for madis2nc config file

[JohnMHenderson]

Hello,

After some trial and error, it's still unclear to me how to set the time_summary settings properly in the madis2nc config file, Madis2NcConfig_default. I would like to compute the max temp over a 24-h window from 0600-0600 UTC each day. I keep getting multiple time windows apparently related to the value of 'step', even while doing simple tests with a much shorter beg..end period that doesn't doesn't overlap two days. I have the flag, grib_code and type parameters set properly. Any help would be much appreciated.

This is the file's contents from the user manual:

time_summary = {
flag = FALSE;
beg = "000000";
end = "235959"
step = 300;
width = 600;
// width = { beg = -300; end = 300; }
grib_code = [ 11, 204, 211 ];
obs_var = [];
type = [ "min", "max", "range", "mean", "stdev", "median", "p80" ];
vld_freq = 0;
vld_thresh = 0.0;
}

Thanks.

[j-opatz]

Hi John,

And thank you for your question. It sounds like there may be an improper setting in your step and width variables for the time_summary dictionary.

Looking at the MET user's guide for the dictionary in question, we find:

Observations may be summarized across the user specified time period defined by the beg and end entries in HHMMSS format. The step entry defines the time between intervals in seconds. The width entry specifies the summary interval in seconds.

Additionally, the guide provides the following example (which is the example you copied):

This example listed above does a 10-minute time summary (width = 600;) every 5 minutes (step = 300;) throughout the day (beg = “000000”; end = 235959”;). The first interval will be from 23:55:00 the previous day through 00:04:59 of the current day. The second interval will be from 0:00:00 through 00:09:59. And so on.

Without seeing your configuration file, your description sounds like the step parameter may be set small enough that it does something similar to the example; centering at the top of the step and so summarizing both the previous day's and day of focus' values. So if I took a first crack at the step and width settings for what you're looking for (max temp over 24 hour period), I'd say you need to adjust the width to be a same 24 hour period, or 86400 (both step and width are in seconds). You can utilize the second example of width with something like

width = {beg = 0; end = 86400}

which says to center the width at the 12 hour mark with 12 hours on each side, eliminating any issue of the width being centered at the step (which would grab the previous day's last 12 hours and your focus day's first 12 hours).

Hopefully this helps you find a successful path through your problem! If it has, please feel free to mark the answer and I'll close the discussion. If you're still running into issues, please reply and upload your configuration file, along with the command you're using to run madis2nc.

Thanks,

-John O.

[JohnMHenderson]

Hi John,

Thanks for your thoughts. Previously, I had set width to 0..86400, plus other shorter windows for testing efficiency, in order to scope out the possible responses. I think a main concern is what value of step should be used when there should not be any stepping happening? That is, I only want one max/min window - the entire width. I've tried 0, 1, 86400 and a very large number for step. These result in either multiple windows (as expected), a hang (really, I think it was churning away still) and an empty output file.
Thanks.
John

[JohnMHenderson]

Hi again John,

I should also state that I'd like to have the time window be the 24-hour window after the valid/file time, so, e.g., 0600 UTC 1 September to 0600 UTC 2 September 2021 would be a file with the date of 20210901_0600.
John

[j-opatz]

John,

Would you be able to upload your config file for MADIS2NC? It would probably help illustrate the issues you're running into.

From the previous discussion, it sounds like you could accomplish the 0600 to 0600 summation by using

beg = "060000";
end = "055959";
step = 86400;
width = {beg = 21600; end = 108000;}

I think by setting your end time to 1 minute prior to the begin time, it avoids an issue where MET thinks your begin and end time is the same, and hence no summation data. The step and width are actually the same (24hours), but you need to tell the width parameter where the center of your summary is (or from a different perspective, where the edges are). The above write up informs MET to start at 0600 (when it should start reading in data) and proceed 24 hours to get to the outer limit (the next 0600) for the summary. The step parameter will then move the next summary to that next 0600 where the previous summary ended, and repeat the process.

Hopefully that's not something that's been tried yet and yields some success. If not, I'll see if some guidance can come from having your configuration file in hand.

Thanks,

John O.

[JohnMHenderson]

Hi John,

Thanks for the guidance. Using your settings, I was able to get an obs max and min file with a valid time at the beginning of the time window, but I'm having problems of course matching with the valid time at the end of the summary window of a corresponding forecast. It doesn't appear that you can set the valid time, unless I'm mistaken.

It would appear from your example that 'beg' and 'end' in the width parameter are relative to 0000 UTC on the current day. Thus, 21600 s and 108000 s correspond to a summary from 0600 UTC on the current day to 0559 UTC on the next day. This setting happens to give a valid time of 0600 UTC on the current day. Interestingly, if I set these values of width to beg=-108000 and end=-21600, I get a valid time of 0600 UTC on the next day, but it's not clear why.

As well, I've played around with trying to set other parameters like vertical level because when I run point_stat with the obs file generated with a matching valid time (via the example above), I then get vertical mismatch errors. It appears that the MADIS2NC config file is only used for time_summary, but again it's not clear what the level should be set for point_stat. The fields obs_hgt and obs_lvl in the obs max/min temp files are missing (e.g., obs_hgt = _, _,).

Madis2NCConfig:
beg = "060001";
end = "060000";
step = 86400;
// width = {beg = 21600; end = 108000;} // gives valid 06Z 20110411
width = {beg = -108000; end= -21600;} // gives valid 06Z 20110412
grib_code = [ 11 ];

PointStatConfig:
obs = {
field = [
{
name = "TMP";
grib_code = [ 11 ];
level = [ "Z002" ];
}
];
}

Any thoughts would be welcomed.

Thanks.

John

[JohnHalleyGotway]

Hi @JohnMHenderson, I wanted to follow up on this issue. I see you have a question about how the time_summary settings translate into the valid times that appear in the output. And I do see how these settings can be confusing!

Let's work through another example to see if that helps clarify. Let's compute 1-hour time summaries every 3 hours during the day.

So we want time summary output for hours 00, 03, 06, 09, 12, 15, 18, and 21. That's specified in the time_summary config section like this:

beg = "00"; // In HH[MMSS] format
end = "21"; // In HH[MMSS] format
step = 3 * 60 * 60; // Defined in seconds: 3 hours * 60 minutes * 60 seconds

We should see time_summary output spaced 3 hours apart in the output.

The next step is defining time windows relative to those output times listed above. And here's the confusing part. You have 2 choices:

1. Define the width of the time window as an integer number of seconds:

width = 60 * 60; // 1-hour width

By convention, this gives you a "symmetric" time summary, i.e. +/- 30 minutes. So the time_summary value for 12Z will contain all observations from 11:30 to 12:30. And 18Z contains all observations from 17:30 to 18:30.

2. Define the width as a dictionary for an "asymmetric" time window:

width = { beg = -3600; end = 0; }

This gives you data for the hour before the timestamp. It defines a retention time window as [ output time + beg, output_time + end ]. So the value reported for 12Z summarizes obs from [12Z - 1 hour, 12Z + 0 hours], which is 11 to 12Z.

width = { beg = 0; end = 3600; }

This gives you data for the hour after the timestamp. So the value at 12Z summaries obs from 12 to 13Z.

So beg/end/step define the valid times that are written to the output... and width defines the time windows to be used relative to those valid times. While you technically could run with neither beg or end set to 0, I doubt that's what you really want. If I'm reading the last message right, I think you want the min/max value to be computed for the previous 24 hours. If that's correct, I'd try using:

beg = "06";
end = "06";
time_step = 24 * 3600;
width = { beg = -24*3600; end = 0; }

[JohnMHenderson]

Hi @JohnHalleyGotway, Thanks as always for your excellent guidance. Your last example seems like the most appropriate way of doing it.

Exploring further: I'm curious if 'beg=06' and 'end=06' here refer to the valid time on subsequent days (and not the same day) because of specification of an asymmetrical 'width' that forces a 24-h window ending at a 06Z (What about a request with a zero-length window?)? It isn't clear to me that 'end=06' would otherwise necessarily refer to the next day. Does the value of the current day (and valid time) come from the first obs file that is input? As you can imagine, to compute the 24-h max/min in this example, I have obs files over a couple of days. If I were to reorder their presentation to the executable, would the idea of current and previous and next change?

More relevant to my current situation, however, is trying to find the subsequent time-averaged obs in point_stat. I was trying to simply use message_type=ADPSFC, which lumps in all surface obs (defined how?) by default, if I recall correctly. However, point_stat now still cannot find the new obs and I can't seem to gain headway on adjusting the vertical levels in either madis2nc or point_stat. The new type output by madis2nc has a suffix that interestingly is not exactly 24 hours (hdr_typ_table="ADPSFC_MIN_235320"), but I'm not sure if this is the current problem.

Any guidance on this aspect would also be much appreciated.

Thanks.

John

[JohnHalleyGotway]

@JohnMHenderson let me point you to the SummaryObs::summarizeObs() function where this logic is implemented.

The function first sorts the observations by valid time (line 153).
Then it defines time intervals based on the first and last valid times (line 161).
The "beg" and "end" times are defined as hours (or hours, minutes, and seconds) of the day. So basically they define the time periods of the day for which you want summarized output. So the code really is using the beginning/ending day/times of the actual observations being processed and then stepping through the requested hours of each day.

Honestly, it's probably more difficult explaining this in words than it is tracing through the logic of the code. If you do get a specific unexpected result in the time summary, please send a sample data file/config file to demonstrate. Then we could run through a debugger to see if there's a problem in the logic.

The specification of the message type is pretty simplistic. The NetCDF output created by pb2nc and madis2nc (for example) include variables named hdr_typ and hdr_typ_table. For example:

ncdump -v hdr_typ out/pb2nc/sample_pb.nc
...
 hdr_typ = 0, 0, 0, 0, 0,

ncdump -v hdr_typ_table out/pb2nc/sample_pb.nc
...
 hdr_typ_table =
  "ADPUPA",
  "AIRCFT",
  "PROFLR",
  "VADWND",
  "ADPSFC",
  "SFCSHP",
  "RASSDA" ;

The hdr_typ integer is just a lookup into the hdr_typ_table strings. So for each obs, we get the string that defines the message_type. In the Point-Stat config file, if you say "message_type = [ "ADPSFC" ];" that'll only use observations identified by that message type in the NetCDF point observation input file.

When using time summaries, you're correct that the time summary information is encoded in that message type. For example:

ncdump -v hdr_typ_table metar_20120409_time_summary.nc
...
 hdr_typ_table =
  "ADPSFC_MIN_030000",
  "ADPSFC_MAX_030000" ;

So take a look at the message types in the MADIS2NC output file and reference use those string in your Point-Stat configuration file:

message_type = [ "ADPSFC_MIN_030000' ];

Hope that helps clarify.

[JohnMHenderson]

Hello again @JohnHalleyGotway,

I had tried tailoring the message_type (including the suffix) to what is present in the madis2nc output file (attached), but with no luck. I've tried a number of different ways to specify/match the vertical levels, both in madis2nc and then in point_stat, but they don't ever seem to do anything. Do you see a configuration (e.g., a value for "level" in combination with name="TMP" and grib_code=[11]) by which I can simply use all the obs in the attached file in point_stat?

[As an aside, I should note that when I use beg=06 or 060000 (and end similarly), I get a valid time at 0600Z, however, with beg=0600 I get a valid time of 0000Z. This likely is because the latter case is not strictly following the HH[MMSS] syntax, though it is still a bit strange.]

Thanks.

John
Tmax-min-20110412_0600.nc.gz

[JohnHalleyGotway]

Thanks for sending the sample output file. FYI, I ran plot_point_obs and see that it's a global dataset:

plot_point_obs Tmax-min-20110412_0600.nc Tmax-min-20110412_0600_plot.ps

To test, I ran point_stat to compare to a GFS file, but I keep getting 0 matched pairs because of a level mismatch:

DEBUG 2: Number of matched pairs   = 0
DEBUG 2: Observations processed    = 9564
...
DEBUG 2: Rejected: level mismatch  = 9564

Looking at the madis2nc output file, I see that madis2nc is reporting the level as bad data in the output:

ncdump -v obs_lvl Tmax-min-20110412_0600.nc
obs_lvl = _, _, _, _, _, ...

And Point-Stat currently has no way of matching a level value of bad data. So I think this is an issue in the time summary output computed by madis2nc.

[JohnHalleyGotway]

FYI, I wrote up this bugfix issue to describe the problem:
dtcenter/MET#1974

Feel free to comment on that issue if you'd like.

[JohnHalleyGotway]

@JohnMHenderson I did some additional testing and have some useful info for you.

The short answer is this. Please try updating your Point-Stat config file like this:

message_type   = [ "ADPSFC_MIN_030000", "ADPSFC_MAX_030000" ];
message_type_group_map = [
   { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET,ADPSFC_MIN_030000,ADPSFC_MAX_030000"; },
...

Let me explain what's happening here. Within Point-Stat and Ensemble-Stat, when verifying against point observations, one important step is making sure that the point observations exist at or within the expected vertical level. Typically that's a pressure level or height level. Just depends on what forecast data you're trying to verify.

However, there is an exception to this rule. If verifying against observations at the surface, we do not need to check the vertical level. The code skips the vertical level validation when verifying against observations with a "surface" message type. In earlier versions of MET, those surface message types were hard-coded to something like "ADPSFC" or "SFCSHP". However in more recent versions, the surface message type are defined by the "SURFACE" entry of the message_type_group_map (as seen above).

By putting those derived time summary message type strings in that list, it should enable Point-Stat to find matched pairs.

Please give that a try and confirm it produces the desired result. If so, I'd recommend that we enhance the processing of the message_type_group_map entries to support wildcards or regular expressions. That way we could use "ADPSFC.*" without needing to explicitly list all time summary types.

[JohnMHenderson]

Hi @JohnHalleyGotway, thanks for the update. I recall this functionality from discussions with you years ago and that was why I thought use of the MADIS surface obs should work, however, I then expected that the bug had negated this ability. I will try your suggestion and let you know. Thanks again.

[JohnMHenderson]

@JohnHalleyGotway This does seem to work. I need to see if the need to explicitly list the time stamp is a complication.

[JohnHalleyGotway]

@JohnMHenderson, great, thanks for confirming. FYI, it does seem tedious to have to explicitly list all time summary message type strings in the Point-Stat message_type_group_map configuration file, so I wrote up this development issue (dtcenter/MET#1974) to make that easier. For example, having the ability to interpret any message type that BEGINS with "ADPSFC" would make things easier.

[JohnHalleyGotway]

If your original question has been answered, feel free to click the "Mark as answer" button on whichever previous response is most useful. And then I'll eventually "Lock" the conversation on this discussion. That's how we encourage users to post new questions to new discussions. And we've found it difficult to keep track of new posts to previously answered discussions.