configuration for using climate data

[sherrieF]

Hi,
I would like to use climate data for in my METplus runs. I have the following in my configuration file.
GRID_STAT_CLIMO_MEAN_FILE_NAME = /scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.1959{valid?fmt=%m%d}
154 GRID_STAT_CLIMO_MEAN_FIELD = URGD
155 GRID_STAT_CLIMO_MEAN_REGRID_METHOD = NEAREST
156 GRID_STAT_CLIMO_MEAN_REGRID_WIDTH = 1
157 GRID_STAT_CLIMO_MEAN_REGRID_VLD_THRESH = 0.5
158 GRID_STAT_CLIMO_MEAN_REGRID_SHAPE = square
159 GRID_STAT_CLIMO_MEAN_TIME_INTERP_METHOD = DW_MEAN
160 GRID_STAT_CLIMO_MEAN_MATCH_MONTH = TRUE
161 GRID_STAT_CLIMO_MEAN_DAY_INTERVAL = 60
162 GRID_STAT_CLIMO_MEAN_HOUR_INTERVAL = 3600
GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = cmean_1d.1959{valid?fmt=%m%d}
GRID_STAT_CLIMO_STDEV_INPUT_TEMPLATE = cmean_1d.1959{valid?fmt=%m%d}

The error I get is
DEBUG 1: Default Config File: /contrib/met/10.0.0/share/met/config/GridStatConfig_default
54 DEBUG 1: User Config File: /contrib/METplus/METplus-4.0.0/parm/met_config/GridStatConfig_wrapped
55 ERROR :
56 ERROR : DictionaryEntry::dict_value() -> bad type
57 ERROR :

I am trying to run on Hera in the directory /home/Sherrie.Fredrick/METplus/gridstat
My log output is in
/scratch2/BMC/gsienkf/Sherrie.Fredrick/C96_replay_winter/metout/logs
I can run my /home/Sherrie.Fredrick/METplus/gridstat/GridStat.conf file just fine without the climo dictionary
included.

Many Thanks,

I can run METplus just fine without the climo dictionary in the GridStat.conf.

[JohnHalleyGotway]

@sherrieF I see you have a question about configuring Grid-Stat via METplus to include the specification of climatology data. Thanks for pointing me to your log files. That really helps me debug the situation. I inspected log output from your last call to Grid-Stat in this file:
metplus.log.20211107222804
These log messages from METplus indicate that you're setting paths to very reasonable values:

11/07 22:28:09.332 metplus INFO: METPLUS_CLIMO_MEAN_DICT=climo_mean = {file_name = ["/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591212"];}
11/07 22:28:09.332 metplus INFO: METPLUS_CLIMO_STDEV_DICT=climo_stdev = {file_name = ["/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591212"];}

And I double-checked to confirm that these files actually do exist. However this log message from Grid-Stat indicates that no climo data is being read:

DEBUG 3: Found 0 climatology mean and 0 climatology standard deviation field(s) for forecast UGRD/P850.

So it looks like we're awfully close. I think the piece we're missing is setting the "climo_mean.field" and "climo_stdev.field" arrays in the config file. So while we're telling Grid-Stat the files from which to read data, we're not telling them what data to read.

Checking in the METplus User's Guide, we should take a look at GRID_STAT_CLIMO_MEAN_FIELD and GRID_STAT_CLIMO_STDEV_FIELD settings. And I did find this example of using the GRID_STAT_CLIMO_MEAN_FIELD.

So I'd recommend updating your METplus config file with something like this:

GRID_STAT_CLIMO_MEAN_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }
GRID_STAT_CLIMO_STDEV_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }

I was trying to find your input "GridStat.conf" file to take a closer look, but was unable to find the directory for that file. Please try adding these config options and let me know how it goes. If it all works, then great. If not, please point me to "GridStat.conf" so I can take a closer look. And we could also loop in @georgemccabe to see if there are more intuitive ways we could setup the specification of climo datasets.

[sherrieF]

I made the following change in my confg file. GRID_STAT_CLIMO_MEAN_FIELD = {name = "UGRD"; level = "P850","P500","P250";} I still get the errors. My config file is in /home/Sherrie.Fredrick/METplus/gridstat*/*GridStat.conf on Hera. Thanks for your help with this. Sherrie

[JohnHalleyGotway]

@sherrieF please try updating your setting to include square brackets around the list of multiple level values. Note, I just modified my previous post (above) to include this detail:

GRID_STAT_CLIMO_MEAN_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }
GRID_STAT_CLIMO_STDEV_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }

Does this change cause any difference in the output?

[sherrieF]

Hi, I did put square brackets around the field name. GRID_STAT_CLIMO_MEAN_FIELD = {name="UGRD"; level=["P850","P500","P250"];} Now I get the following error:ERROR : DictionaryEntry::dict_value() -> bad type Thanks, Sherrie

[JohnHalleyGotway]

Sherrie, checking in the log file I see a problem in the handling of the double-quotes.
Here's what you put in the GridStat.conf file:

GRID_STAT_CLIMO_MEAN_FIELD = {name="UGRD"; level=["P850","P500","P250"];}

And here's what showed up in the METplus log file:

11/08 18:22:02.874 metplus INFO: METPLUS_CLIMO_MEAN_DICT=climo_mean = {file_name = ["/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591212"];field = ["{name=\"UGRD\"; level=[\"P850\",P500,P250];}"];regrid = {method = NEAREST;width = 1;vld_thresh = 0.5;shape = SQUARE;}time_interp_method = DW_MEAN;match_month = TRUE;day_interval = 60;hour_interval = 3600;}

Note that the quoting of the level values is inconsistent.

@georgemccabe can you please take a look at this issue with double quotes being applied? Is this a known issue or a new one?

[JohnHalleyGotway]

Sherrie, I've talked with @georgemccabe and @j-opatz and realize that some changes were made to the development version of METplus for the next release to support this. However, rather than asking you to switch to using a development version, I'd like to try an alternative approach. Can you please edit your GridStat.conf file as follows:

1. Comment out the settings I had you add by putting a # symbol in front of them:

#GRID_STAT_CLIMO_MEAN_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }
#GRID_STAT_CLIMO_STDEV_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }

2. Add the config overrides setting below:

GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = fcst; climo_stdev = fcst;

And then try rerunning. Here's what this is doing... Rather than explicitly defining the climo fields directly, we can take advantage of the fact that we want the exact same name/level settings from the climo files that we want from the forecast files. The setting above should initialize the field list for climo_mean and climo_stdev to what we already have defined in the fcst dictionary.

Does that produce any different results?

[sherrieF]

Good Afternoon, I made the change in my config file. I have GRID_STAT_CLIMO_MEAN_FILE_NAME = /scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.1959{valid?fmt=%m%d} GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = fcst; climo_stdev = fcst; The error I now get is: WARNING: GRIB2 records not found in input file '/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591203' WARNING: ERROR : ERROR : Met2dDataFileFactory::new_met_2d_data_file() -> error opening file "/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591203" ERROR : Unfortunately my model output is in GRIB2 format. My era5 data which I an using for the OBS has U for the field rather than UGRD. Cheers, Sherrie

[JohnHalleyGotway]

Sherrie, well the good news is that this is progress at least! Looking in your METplus log file, I see a rather curious log message:

g2_info: can only decode GRIB edition 2.WARNING: 
WARNING: GRIB2 records not found in input file '/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591212'
WARNING: 
ERROR  : 
ERROR  : Met2dDataFileFactory::new_met_2d_data_file() -> error opening file "/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591212"
ERROR  :

But I think I know what's going on. Your forecast dictionary includes "file_type = GRIB2;" But when we copy over those settings using climo_mean = fcst; and climo_stdev = fcst; we're also copying over that file_type setting! So we're telling Grid-Stat to process the climo data as if they were GRIB2 files. And that's the source of the error you're seeing.

So let's shift gears again, and have you try running the latest development version of METplus on Hera instead. Please take a look on this page. Under "NOAA Machines -> Hera" you'll see instructions for loading the METplus 4.1-beta3 release. You could try pointing to that version... after updating your config file to uncomment the old settings and comment out the new one:

GRID_STAT_CLIMO_MEAN_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }
GRID_STAT_CLIMO_STDEV_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; }
#GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = fcst; climo_stdev = fcst;

Thanks for bearing with me. Hopefully us working through the details will provide good feedback for METplus development.

[sherrieF]

Hi John, I tried running /contrib/METplus/METplus-4.1.0-beta3/ush/run_metplus.py. I had the following in my config file GRID_STAT_CLIMO_MEAN_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; } The error I got was ERROR : ERROR : yyerror() -> syntax error in file "/scratch2/BMC/gsienkf/Sherrie.Fredrick/C96_replay_winter/metout/tmp/met_config_237939_0" ERROR : ERROR : line = 72 ERROR : ERROR : column = 42 ERROR : ERROR : text = "P500" ERROR : ERROR : ERROR : climo_mean = {file_name = ["/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591203"];field = [{name="UGRD"; level= ["P850",P500,P250]; }];} Cheers, Sherrie

…

On Mon, Nov 8, 2021 at 4:20 PM johnhg ***@***.***> wrote: Sherrie, well the good news is that this is progress at least! Looking in your METplus log file, I see a rather curious log message: g2_info: can only decode GRIB edition 2.WARNING: WARNING: GRIB2 records not found in input file '/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591212' WARNING: ERROR : ERROR : Met2dDataFileFactory::new_met_2d_data_file() -> error opening file "/scratch1/NCEPDEV/global/glopara/fix/fix_verif/climo_files/cmean_1d.19591212" ERROR : But I think I know what's going on. Your forecast dictionary includes "file_type = GRIB2;" But when we copy over those settings using climo_mean = fcst; and climo_stdev = fcst; we're also copying over that file_type setting! So we're telling Grid-Stat to process the climo data as if they were GRIB2 files. And that's the source of the error you're seeing. So let's shift gears again, and have you try running the latest development version of METplus on Hera instead. Please take a look on this page <https://dtcenter.org/community-code/metplus/metplus-4-1-existing-builds>. Under "NOAA Machines -> Hera" you'll see instructions for loading the METplus 4.1-beta3 release. You could try pointing to that version... after updating your config file to uncomment the old settings and comment out the new one: GRID_STAT_CLIMO_MEAN_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; } GRID_STAT_CLIMO_STDEV_FIELD = { name = "UGRD"; level = ["P850","P500","P250"]; } #GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = fcst; climo_stdev = fcst; Thanks for bearing with me. Hopefully us working through the details will provide good feedback for METplus development. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1242 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK2TAEIDZ5KNHVFU3BJFAVTULBLMXANCNFSM5HOKWO2A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[sherrieF]

Hi again John, I tried the following to get rid of the error. This did not give me an error, but the message in the log file is Found 0 climatology mean and 0 climatology standard deviation field(s) for forecast UGRD/P850. GRID_STAT_CLIMO_MEAN_FIELD = {name="UGRD"; level=["P850","P500","P250" Cheers, Sherrie

[georgemccabe]

Hi @sherrieF , I logged onto Hera to do some testing and it looks like the climo variables don't properly format a list of field levels. I was able to get your use case to run successfully by separating out each level into its own entry:

GRID_STAT_CLIMO_MEAN_FIELD = {name="UGRD"; level="P850";},{name="UGRD"; level="P500";}, {name="UGRD"; level="P250";}

I created GitHub issue #1247 to fix this bug.

[JohnHalleyGotway]

Thanks @georgemccabe for taking a look!

Sherrie, can you please try out the different formatting that worked for George?

GRID_STAT_CLIMO_MEAN_FIELD = {name="UGRD"; level="P850";},{name="UGRD"; level="P500";}, {name="UGRD"; level="P250";}
GRID_STAT_CLIMO_STDEV_FIELD = {name="UGRD"; level="P850";},{name="UGRD"; level="P500";}, {name="UGRD"; level="P250";}

I assume we'll be able to include a fix for #1247 in the next release.

Thanks,
John

[sherrieF]

Hi John, Yep, I copied your GRID_STAT_CLIMO_MEAN_FIELD into my config file and it worked. So many thanks for helping with this. I really wanted to use METplus. Sherrie

[JohnHalleyGotway]

Thanks for bearing with us. We appreciate your patience!
We'll have some more discussions internally about how to make the configuration of climo data more intuitive.

For this discussion, the next step is to go to GitHub and select one of the responses in this chain as being a good "Answer". And then we'll eventually "lock" the conversation to prevent future posts. That's our way of encouraging users to post new questions to new discussions. And we've found that it's difficult to keep track of new posts to previously "answered" discussions.

Thanks,
John

[JohnHalleyGotway]

Thanks Sherrie. @georgemccabe is updating #1247 to include all the tweaks we think are needed for the next release. I'll go ahead and lock this conversation now. Thanks.