Different RMSEs produced by EnsembleStat and by GenEnsProd/GridStat

[wuxx66]

I have used EnsembleStat to produce RMSE from ECNT. Then I used GenEnsProd to produce a netCDF file, then used GridStat to generate CNT. In CNT, there is RMSE too. The 2 RMSEs are different. I am wondering what are the differences?

Could you please help this?

Thank you

Xiaoxi

[JohnHalleyGotway]

Hello Xiaoxi, I see you have a question about the Root-Mean-Squared-Error (RMSE) reported by both the Ensemble-Stat and Grid-Stat tools in MET. You're correct to point out that RMSE appears in column 31 of the ECNT line type written by Ensemble-Stat and column 78 of the CNT line type written by Grid-Stat.

The RMSE value reported by Ensemble-Stat is for the ensemble mean (derived on the fly from the members) and is computed using the observation values provided. I see that you also ran Gen-Ens-Prod to compute and write the ensemble mean field. Then you configured Grid-Stat to compare the Gen-Ens-Prod output ensemble mean to the same observation values you provided to Ensemble-Stat.

And you note that these 2 different methods produce different RMSE values. I certainly agree with you that they should be identical... but only if all the configuration options are completely consistent.

As a sanity check, I repeated this test using sample data provided in the MET tar file. Here's the config files, run script, and output:
METplus_2637.tar.gz

With this data, the RMSE values really are identical:

cat ensemble_stat_20100101_120000V_ecnt.txt | awk '{print $31}'
RMSE
10.75308
cat grid_stat_APCP_24_240000L_20100101_120000V_cnt.txt | awk '{print $78}'
RMSE
10.75308

I'd recommend checking the numbers in the TOTAL column of your output files. In my case, both are the same:

TOTAL
15480

Since I'm computing statistics over exactly the same set of grid points and values, I get the same RMSE output.

Ultimately, I'd recommend that you compare the values in the first 25 columns of the ECNT and CNT lines. All .stat line types have the same common header columns. If something differs, it should stand out. For example, perhaps you've used different masking regions (VX_MASK) or different interpolation methods (INTERP_MTHD), or something else?

If all the header columns do match, but the TOTAL column does not, check the skip_const flag in your Ensemble-Stat config file. If that's set to true and you're verifying precip, you'd be tossing out several points that have 0 precip in all the inputs. The would result in fewer matched pairs in the ECNT line type when compared to the CNT line type. And a different set of matched pairs will certainly produce a different RMSE result.

Please let me know how your investigation goes. If you have more questions, please feel free to send config files, log files, and/or output files to illustrate.

[wuxx66]

Hello John,
Thank you for your detailed reply. I have compared my .stat files. The first 25 columns do match and total numbers of columns are same too, but RMSEs are still different. I could not work out what has caused the difference. Could you help me identify the problem? I am actually verifying T850 against its own analysis. I have sent the config file, log files and output files to ftp and they can be found in wu_data?

Thank you so much

Xiaoxi

[JohnHalleyGotway]

Please follow these instructions to upload data to our anonymous ftp site. And then please post again here to let me know that there are files I should grab.

[JohnHalleyGotway]

Xiaoxi, thanks for sending along this information. I spent some time stepping through the details, but unfortunately, did not uncover an obvious reason why you're getting different RMSE values from Ensemble-Stat and Grid-Stat.

Along that way I did note the following:

• Path 1: Run Ensemble-Stat directly and generate RMSE in the ECNT line type.
  • Produces: TOTAL = 11280 and RMSE = 0.7873
• Path 2: Run Gen-Ens-Prod followed by Grid-Stat to generate RMSE in the CNT line type.
  • Produces: TOTAL = 11280 and RMSE = 0.88553
• The first 25 header columns of the ECNT and CNT lines are identical, other than the ALPHA column being 0.05 for CNT, but that's expected and is not a source of this difference.
• All Ensemble-Stat, Gen-Ens-Prod, and Grid-Stat inputs are regridded to the same lat/lon grid (latlon 240 47 -90 -360 1.5 1.5) with the same method, shape, and width.
• Checking for WARNING messages, I do this in the log output 3 times:

WARNING: RegridInfo::validate() -> Resetting regridding width from 2 to 1 for interpolation method "AW_MEAN".

• This is coming from Grid-Stat. Since you haven't set GRID_STAT_REGRID_WIDTH = 1, it uses the default value of 2... but then resets it to 1 because you're using AW_MEAN. Adding that GRID_STAT_REGRID_WIDTH = 1 option would make the warning go away. But that isn't causing the RMSE difference.

The TMP_P850_ENS_MEAN field in gen_ens_prod_20230307_000000V_ens.nc is the ensemble mean field computed by Gen-Ens-Prod. The TMP_P850_ENS_MEAN_FULL field in ensemble_stat_20230307_000000V_orank.nc is the ensemble mean field computed by Ensemble-Stat. I ran PCP-Combine to compute their difference:

pcp_combine -subtract ensemble_stat_20230307_000000V_orank.nc 'name="TMP_P850_ENS_MEAN_FULL"; level="(*,*)";' gen_ens_prod_20230307_000000V_ens.nc 'name="TMP_P850_ENS_MEAN"; level="(*,*)";' diff.nc

And the fields are identical. So there's no difference in the ensemble mean values being computed.

Another possibility could be in the gridded observation data. We already have the gridded observation data output from Ensemble-Stat (TMP_P850_OBS_FULL), but we're missing that same output from Grid-Stat. Please update your METplus configuration by setting:

GRID_STAT_NC_PAIRS_FLAG_LATLON = TRUE
GRID_STAT_NC_PAIRS_FLAG_RAW = TRUE
GRID_STAT_NC_PAIRS_FLAG_DIFF = TRUE

And then rerun.

We'll want to compare the gridded obs field reported by Ensemble-Stat to the same from Grid-Stat. Perhaps we'll find some difference there and that will provide more clues.

[wuxx66]

Hi John,
Thank you very much for looking into this issue. I have made the changes in the config file as you suggested.

GRID_STAT_NC_PAIRS_FLAG_LATLON = TRUE
GRID_STAT_NC_PAIRS_FLAG_RAW = TRUE
GRID_STAT_NC_PAIRS_FLAG_DIFF = TRUE

I have re-run the job, but unfortunately RMSEs are still different.

I have updated the config file and results in the ftp site under "wu_data"

[wuxx66]

Hi John,
We have identified the issue. In the config file, there is a setting "ENSEMBLE_STAT_GRID_WEIGHT_FLAG = COS_LAT", we added "GRID_STAT_GRID_WEIGHT_FLAG = COS_LAT" in GridStat part. Now RMSEs from both EnsembleStat and GridStat are identical.

[JohnHalleyGotway]

Whew! Glad you figured that out. And thanks for following up to let me know.

Yes, there are lots of options for how the data can be processed. Defining grid weights is one such option. Others include applying a convert(x) function to change units or using the censor_thresh and censor_val options to discard some points based on their values. And there are others.

In an ideal world, we'd make the output written by the MET tools completely self-describing. So you would have been able to compare the Grid-Stat output metadata to the Ensemble-Stat output metadata and that configuration difference would be obvious. However when writing ASCII .stat output files, it really isn't realistic to make them fully self-describing. We're limited to the 20+ existing header columns that are common to all the output line types and adding more isn't feasible.

Instead, let me call your attention to the desc configuration option. It can be specified separately for each individual verification task. It's just a free-form text string that gets written to the output DESC header column.

Let's say you've configured Grid-Stat to compute the same stats twice, once with no grid weighting and a second time with cosine latitude weighting. In that case, you should set the desc column to something like COS_LAT_WEIGHTING for one and NO_WEIGHTING for the other. Basically, that gives you a way to distinguish the statistics that have been computed with different filtering/processing options. Without it, you wouldn't be able to tell which is which.

[wuxx66]

Hi John, thank you very much for giving us many clues and helping us figure out the issue. It is a very good idea to set the desc column in each task, so we are able to tell which is which.