Error reading netCDF file GridStat

[jtv108-wx]

Hello,

I'm having issues reading a netCDF file into GridStat. I'm getting this error in my output file,...

--- NcCfFile::get_grid_from_lat_lon_vars() -> lon[367]=-112.672 lon[368]=-112.644 0.00133514 > 0.00133324
--- NcCfFile::get_grid_from_lat_lon_vars() -> MET can only process Latitude/Longitude files where the longitudes are evenly spaced (dlon=0.0266647, delta[368]=0.0279999)
--- NcCfFile::get_grid_from_lat_lon_vars() -> Please check the input data is the lat/lon projection

I believed the issue originates from the original netCDF file. From searching other discussion threads I believed the issue was similar to this (#1931). I proceeded to writing a python script to manually read the original netCDF file and set the correct file attributes. Then I ran pcp_combine with python_embedding (using the python code) using the following command...

--- pcp_combine -add PYTHON_NUMPY $Output_filename -field 'name="$Path_to_ascii_numpy_script.py $Path_to_nc_file $nc_file_var";'

The pcp_combine command above successfully runs and outputs a netCDF file. However, when I pass this output netCDF file into GridStat I get the evenly spaced lat/lon error (see above).

Any assistance would be greatly appreciated!

Thanks.

[CPKalb]

Hi:

It sounds like the error is occurring when reading the output from Pcp-Combine into Grid-Stat? Would you be able to send a sample netCDF file so that we could take a closer look at this file? Instructions for sending data can be found here.

Thanks,
Christina

[jtv108-wx]

Yes, that was the original file. Sorry for the confusion. I was hoping I may have missed something when dealing with the original file that would just let me read the original file directly into Grid-Stat (without the Pcp-Combine step needed). But yes, the "evenly spaced lat/lon" error I was getting was a result of reading the Pcp-combine (with python embedding) output into Grid-Stat. I'll send the Pcp-combine output file over right now.

Thanks,
Jake

[CPKalb]

Ah, I didn't realize you were wanting to omit the Pcp-Combine step. Would you be able to post the python embedding script that you are using for Pcp-Combine here? You will likely need to give it a .txt extension to be able to post. Alternatively, you could put it on the ftp site.

Thanks,
Christina

[jtv108-wx]

Well if it is not necessary to be able to read the file (into Grid-Stat) I wouldn't have to use the Pcp-Combine step. I was trying the fix I found here (#1931) but I'm not sure if that actually applies to my case or files. But okay, I'll just add the python embedding script to the ftp site.

Thanks,
Jake

[CPKalb]

In the issue referenced, they used Pcp-Combine specifically because they didn't want to use python embedding with Grid-Stat (they were getting inconsistent VL1L2 output with python embedding due to a bug we had in MET). If you are okay with reading the file into Grid-Stat using python embedding, then the Pcp-Combine step should not be necessary here.

The original file isn't in one of the netCDF formats that MET can read directly.

Christina

[jtv108-wx]

Ah okay, sounds great. Yes, if possible I'd like to read the original netCDF file right into Grid-Stat (with python embedding). I'm very new to the python embedding component with MET/METplus.

Jake

[CPKalb]

Hi Jake:

I am able to get the data file to go directly into Grid-Stat using python embedding with MET version 11.1.0. Are you running this using a METplus configuration file or directly in MET? I tested directly in MET. Assuming this is your forecast data, you would want to set the following to get it to work:

In the MET configuration file, for the forecast variable set: name = "read_ascii_numpy.py 01_test.nc refc";
The level variable is omitted here in the configuration. When you run Grid-Stat, you would want to do the following on the command line:
grid_stat PYTHON_NUMPY observation_input_file GridStatConfigurationFile

If you are running from METplus, the setup would be a bit different, and I would need to see your METplus configuration to help get it up and running.

When using python embedding, it's always a good idea to run a test using plot_data_plane to see if the data is correctly read into MET. I did this using the following command:
plot_data_plane PYTHON_NUMPY test_nc.ps 'name="read_ascii_numpy.py 01_test.nc refc";'

Here is the resulting image:

The image created by MET's plot_data_plane is flipped compared to what I see when using ncview, which looks like the following:

As you can see, in ncview, the precipitation is in the upper right corner of the image whereas in the plot_data_plane output, it's in the lower right. If the ncview version is correct, then you would want to change line 27 in your script to have the following 2 lines:
data = input_nc.variables[data_name][:]
met_data = data[::-1].copy()

With that change, I get an output image that matches what is displayed in ncview:

Also, I think the date format you have in the python embedding script may be incorrect. If the date of your data is May 1, 2023, then the init and valid should look like this:
'valid': '20230501_010000',
'init': '20230501_000000',

I've attached the python embedding script here with the changes I made.
read_ascii_numpy.py.txt

Christina

[jtv108-wx]

Thank you for the help. I am wondering if there is also an issue with the lat/lon data being read by MET as I'm fairly sure the data is suppose to be over the ECONUS (maybe a -360 type issue?).

I am planning on running from METplus. I moved the conf file I use over to you to try. The command I use to run METplus is...

/usr/bin/python3.6 /usr/local/met10/METplus-4.0.0/ush/run_metplus.py -c $PATH_to_conf_file -c _$PATH_to_metplus_user.conf

Okay, I'll make those grid flipping changes once I confirm the location of the grid over the CONUS. Ah, you are correct on the date format issue, I'll make that change as well.

Jake

[CPKalb]

Hi Jake:

This is an example of how to set up a METplus configuration file when using python embedding in Grid-Stat. I took your configuration file and made the changes that I think need to be made for this to work. However, I can't actually test the configuration since I don't have your observations and mask file. You may need to give the full path to your python script or to the input netCDF file in FCST_VAR1_NAME, depending on where you are running METplus and where the data/script is located.
REFC_TEST01_GridStat.conf.txt

For the netCDF file, if the grid is supposed to be over the CONUS, then most likely something is not correct in the grid specification.

Christina

[CPKalb]

I notice in the netCDF file that it states "2023-08-23T14:14 GRIB to CDM+CF via cfgrib-0.9.10.1/ecCodes-2.22.1". GRIB/GRIB2 files can be easily read into MET so if you have the originals, it may be easier just to read them in as GRIB files.

Christina

[jtv108-wx]

Hi Christina,

Thank you for all of your help. I'll try setting up the conf file changes you attached for the METPlus python embedding run (with Grid-Stat). Ah, okay I'll double the grid specifications as well.

Okay, I'll see if I can track down the original GRIB files and try those as well.

Jake

[CPKalb]

Hi Jake:

I ran a test in plot_data_plane using the grid specified in the new file, and the data came out as below:

The forecast data is aligned with the grid boundaries, but I notice that my domain is shorter on the top and longer on the bottom. In your output from Grid-Stat, the domain is longer on top and shorter at the bottom. Do you know what the domain should look like? Should the longer side be on the top or on the bottom?

I also ran a test in Grid-Stat, however, I don't have your observation file or mask file. When I run Grid-Stat without the observation or mask, the output looks like the following:

It seems that your grid specification is different from the one I have, which is causing the domains to look different, and likely the cause of the misalignment in domain edge.

Christina

[CPKalb]

As a secondary check, I loaded the data file in IDV to try and view reflectivity with the United States as a background. The resulting image looks like this:

In comparison to the MET output, the areas of precipitation are farther North and East in the IDV image. IDV domains don't always display exactly as they do in MET, but the precipitation should be in a similar location. This suggests that the grid specification (lat_pin, lon_pin, lon_orient, scale_lat, etc.) may still be incorrect. MET doesn't read the grid specification in the netCDF file when using python embedding. So, if you know what these numbers should be, this can be fixed inside the read_ascii.py script.

I tried setting lat_pin = 25.7478095952292 and lon_pin = 255.027346249833 in read_ascii.py, and this is the resulting image:

It better matches what is seen in IDV for precipitation. However, these numbers are guesses. It would be good to check all the numbers in the grid specification to verify that everything is correct.

Christina

[jtv108-wx]

Hi Christina,

Thanks for taking the time to look into this further. Let me double check the grid settings and get a few more sample plots. I'll also need to double check the values with the original file creator. Thanks again for all of the help.

Jake

[CPKalb]

Hi Jake:

Here is a bit more information on some of the lambert conformal variables which will hopefully help get the right grid specification. lat_pin and lon_pin are the reference latitude and longitude. These could be the lower-left corner, the midpoint, or another location. The The x_pin and y_pin values define the offset from the lower-left corner to the location of the reference lat/lon values. In your grid, x_pin and y_pin are set to 0, which means that lat_pin and lon_pin should be the lower left corner points of the grid. For lambert conformal grids, the lower left point of the grid are not always the minimum latitude and longitude. lon_orient is the orientation longitude, which is the meridian of longitude that’s parallel to one of the vertical grid directions.

Christina

[CPKalb]

Hi Jake:

I'm following up on this discussion. Were you able to get this working?

Chrisinta

[jtv108-wx]

Hi Christina, Sorry for the very delayed response. This email got buried in my inbox and then I have been away due to an ongoing medical issue. But yes, I was able to get the files read and processed from METplus. Thank you so much for all of your help. Thanks, Jake

…

________________________________ From: Christina Kalb ***@***.***> Sent: Wednesday, March 27, 2024 5:39 PM To: dtcenter/METplus ***@***.***> Cc: Vancil, Jake T. ***@***.***>; Author ***@***.***> Subject: Re: [dtcenter/METplus] Error reading netCDF file GridStat (Discussion #2511) Hi Jake: I'm following up on this discussion. Were you able to get this working? Chrisinta — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/dtcenter/METplus/discussions/2511*discussioncomment-8934015__;Iw!!GNU8KkXDZlD12Q!_GV9Dh61hcQb0HvMx_HgiFjGRozfMOFjvCRL751faqj6GBnCroiNWwtgJ6Gy6RzPuMgEt_yXrfgdREodAtf3F5gwBQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A7X6A7IIQH6VTGDSERUKMPLY2NDDDAVCNFSM6AAAAABEJ46P3GVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSMZUGAYTK__;!!GNU8KkXDZlD12Q!_GV9Dh61hcQb0HvMx_HgiFjGRozfMOFjvCRL751faqj6GBnCroiNWwtgJ6Gy6RzPuMgEt_yXrfgdREodAtcLXxLXqA$>. You are receiving this because you authored the thread.Message ID: ***@***.***>