help with CF compliance #1107

Wesley-J-Davis · 2021-08-24T15:22:25Z

Wesley-J-Davis
Aug 24, 2021

First, I've done my best to edit my metadata in accordance with the CF compliance docs at:
http://cfconventions.org/Data/cf-standard-names/77/build/cf-standard-name-table.html
http://cfconventions.org/cf-conventions/cf-conventions.html#units
https://pumatest.nerc.ac.uk/cgi-bin/cf-checker.pl?cfversion=auto

Here's an example of my metadata:

netcdf cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h {
dimensions:
ens = 16 ;
tau = 44 ;
lat = 73 ;
lon = 144 ;
variables:
float cloud_area_fraction(ens, tau, lat, lon) ;
cloud_area_fraction:long_name = "cloud area fraction" ;
cloud_area_fraction:units = "fraction of 1" ;
int ens(ens) ;
ens:long_name = "ensemble member" ;
float tau(tau) ;
tau:units = "hours" ;
tau:long_name = "time relative to initialization" ;
float lat(lat) ;
lat:units = "degrees_north" ;
lat:long_name = "latitude" ;
float lon(lon) ;
lon:units = "degrees_east" ;
lon:long_name = "longitude" ;
double time(tau) ;
time:units = "hours since 1800-1-1 00:00:0.0" ;
time:long_name = "time relative to reference date" ;
int valid_mem(ens) ;
valid_mem:long_name = "valid member" ;

// global attributes:
:model = "ESPC" ;
:model_grid = "GLBb0.08_T0359_NHC" ;
:model_expt = "17.1" ;
:level_type = "sky_cvr" ;
:level = "0000.0" ;
:var = "cloud_area_fraction" ;
:ens = "e016" ;
:grid_dimensions = "glob144x73" ;
:time_resolution = "daily_avg_12Z_12h" ;
:field_type = "fcstfld" ;
:grid_mapping_name = "latitude_longitude" ;
:regrid_method = "bilinear" ;
:init_time_dtg = "2020051012" ;
:init_time_ref_1800 = 1931604LL ;
:comments = "Matthew Janiga ([email protected])" ;
:date_created = "2021-08-24 14:19:06.733527" ;

Thank you!

Answered by georgemccabe

Aug 25, 2021

It looks like plot_data_plane is interpreting the data as a MET-generated NetCDF file instead of a CF-compliant NetCDF file. I would recommend setting the file type to tell the tool what type of file is being provided.

plot_data_plane cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc test.nc 'name="cloud_area_fraction"; level="(0,0,*,*)"; file_type=NETCDF_NCCF;' -v 4

Let me know if that doesn't work.

View full answer

Wesley-J-Davis · 2021-08-24T15:29:56Z

Wesley-J-Davis
Aug 24, 2021
Author

I tried plot_data_plane using the following format:

plot_data_plane cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc test.nc 'name="cloud_area_fraction"; level="(0,0,,)";' -v 4
DEBUG 1: Opening data file: cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".
ERROR :
ERROR : plot_data_plane -> file "cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc" not a valid data file
ERROR :

Checking the file in the checker at: https://pumatest.nerc.ac.uk/cgi-bin/cf-checker.pl?cfversion=auto gives me:

CHECKING NetCDF FILE: /tmp/27024.nc

WARN: Cannot determine CF version from the Conventions attribute; checking against latest CF version: CF-1.8
Using CF Checker Version 4.1.0
Checking against CF Version CF-1.8
Using Standard Name Table Version 77 (2021-01-19T13:38:50Z)
Using Area Type Table Version 10 (23 June 2020)
Using Standardized Region Name Table Version 4 (18 December 2018)

WARN: (2.6.1): No 'Conventions' attribute present

Checking variable: cloud_area_fraction

Checking variable: ens

WARN: (3.1): units attribute should be present

Checking variable: tau

Checking variable: lat

Checking variable: lon

Checking variable: time

Checking variable: valid_mem

INFO: (3.1): No units attribute set. Please consider adding a units attribute for completeness.

ERRORS detected: 0
WARNINGS given: 3
INFORMATION messages: 1

1 reply

Wesley-J-Davis Aug 24, 2021
Author

I tried plot_data_plane using the following format:

plot_data_plane cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc test.nc 'name="cloud_area_fraction"; level="(0,0,,)";' -v 4
DEBUG 1: Opening data file: cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".
ERROR :
ERROR : plot_data_plane -> file "cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc" not a valid data file
ERROR :

Checking the file in the checker at: https://pumatest.nerc.ac.uk/cgi-bin/cf-checker.pl?cfversion=auto gives me:

CHECKING NetCDF FILE: /tmp/27024.nc

WARN: Cannot determine CF version from the Conventions attribute; checking against latest CF version: CF-1.8
Using CF Checker Version 4.1.0
Checking against CF Version CF-1.8
Using Standard Name Table Version 77 (2021-01-19T13:38:50Z)
Using Area Type Table Version 10 (23 June 2020)
Using Standardized Region Name Table Version 4 (18 December 2018)

WARN: (2.6.1): No 'Conventions' attribute present

Checking variable: cloud_area_fraction

Checking variable: ens

WARN: (3.1): units attribute should be present

Checking variable: tau

Checking variable: lat

Checking variable: lon

Checking variable: time

Checking variable: valid_mem

INFO: (3.1): No units attribute set. Please consider adding a units attribute for completeness.

ERRORS detected: 0
WARNINGS given: 3
INFORMATION messages: 1

I have a .txt file that is the contents of an ncdump, but for some reason it won't let me upload on here.

georgemccabe · 2021-08-25T15:19:22Z

georgemccabe
Aug 25, 2021
Maintainer

It looks like plot_data_plane is interpreting the data as a MET-generated NetCDF file instead of a CF-compliant NetCDF file. I would recommend setting the file type to tell the tool what type of file is being provided.

plot_data_plane cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc test.nc 'name="cloud_area_fraction"; level="(0,0,*,*)"; file_type=NETCDF_NCCF;' -v 4

Let me know if that doesn't work.

4 replies

Wesley-J-Davis Aug 26, 2021
Author

plot_data_plane cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc test.nc 'name="cloud_area_fraction"; level="(0,0,*,*)"; file_type=NETCDF_NCCF;' -v 4 DEBUG 1: Opening data file: cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF". DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1800-1-1 00:00:0.0" DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1800-1-1 00:00:0.0" as a reference time of 18000101_000000 and 3600 second(s) per time step. DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable. DEBUG 4: NcCfFile::open() -> could not extract init time from file name. DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF". DEBUG 4: NcCfFile::getData() -> setting the unset init time to the valid time of 20200511_120000. DEBUG 4: DEBUG 4: Data plane information: DEBUG 4: plane min: 0 DEBUG 4: plane max: 0.99975 DEBUG 4: valid time: 20200511_120000 DEBUG 4: lead time: 000000 DEBUG 4: init time: 20200511_120000 DEBUG 4: accum time: 000000 DEBUG 1: Creating postscript file: test.nc DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-9.1/met-9.1/share/met/map/country_data DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-9.1/met-9.1/share/met/map/usa_state_data DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-9.1/met-9.1/share/met/map/major_lakes_data

At first glance, looks like it works. But when I try to view it using ncview:

ncview test.nc ncview: /usr/local/Anaconda3-2020.02/lib/libuuid.so.1: no version information available (required by /usr/lib64/libSM.so.6) Ncview 2.1.5 David W. Pierce 18 Mar 2015 http://meteora.ucsd.edu:80/~pierce/ncview_home_page.html Copyright (C) 1993 through 2015, David W. Pierce Ncview comes with ABSOLUTELY NO WARRANTY; for details typencview -w'.
This is free software licensed under the Gnu General Public License version 3; type `ncview -c' for redistribution details.

ncview: can't recognize format of input file test.nc
`

georgemccabe Aug 26, 2021
Maintainer

The output from plot_data_plane is a postscript file. You should name the output file test.ps instead of test.nc. You can then convert the ps file to a png if you need with the convert command:

convert -rotate 90 test.ps test.png

If the data in the image lines up with the map outlines properly, then MET can read the data.

Wesley-J-Davis Aug 26, 2021
Author

oh, I should have known that. thanks!

I just used ghostview to open it up and it passes the smell test. Thank you!

Looks like the issue ended up being how I was entering file_type= into the command. I had been putting the NETCDF_NCFF in quotes when I didn't need to.

georgemccabe Aug 26, 2021
Maintainer

Great to hear! Yes, the NETCDF_NCCF text is a keyword recognized by the MET tools so it should not be surrounded by quotes. Please create a new discussion topic if you run into any other issues. Thanks!

hsoh-u · 2021-08-25T16:15:58Z

hsoh-u
Aug 25, 2021
Collaborator

The report of CF -checking:

The "Conventions" global attribute is missing, and the CF checker checked it against CF-1.8 convension.
MET requires the "Conventions" global attribute which begins with "CF-" to be identified as CF-complaint NetCDF file.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

help with CF compliance #1107

{{title}}

Replies: 3 comments 5 replies

{{title}}

{{title}}

CHECKING NetCDF FILE: /tmp/27024.nc

Checking variable: cloud_area_fraction

Checking variable: ens

Checking variable: tau

Checking variable: lat

Checking variable: lon

Checking variable: time

Checking variable: valid_mem

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

help with CF compliance #1107

Wesley-J-Davis Aug 24, 2021

Replies: 3 comments · 5 replies

Wesley-J-Davis Aug 24, 2021 Author

CHECKING NetCDF FILE: /tmp/27024.nc

Checking variable: cloud_area_fraction

Checking variable: ens

Checking variable: tau

Checking variable: lat

Checking variable: lon

Checking variable: time

Checking variable: valid_mem

Wesley-J-Davis Aug 24, 2021 Author

CHECKING NetCDF FILE: /tmp/27024.nc

Checking variable: cloud_area_fraction

Checking variable: ens

Checking variable: tau

Checking variable: lat

Checking variable: lon

Checking variable: time

Checking variable: valid_mem

georgemccabe Aug 25, 2021 Maintainer

Wesley-J-Davis Aug 26, 2021 Author

georgemccabe Aug 26, 2021 Maintainer

Wesley-J-Davis Aug 26, 2021 Author

georgemccabe Aug 26, 2021 Maintainer

hsoh-u Aug 25, 2021 Collaborator

Wesley-J-Davis
Aug 24, 2021

Replies: 3 comments 5 replies

Wesley-J-Davis
Aug 24, 2021
Author

Wesley-J-Davis Aug 24, 2021
Author

georgemccabe
Aug 25, 2021
Maintainer

Wesley-J-Davis Aug 26, 2021
Author

georgemccabe Aug 26, 2021
Maintainer

Wesley-J-Davis Aug 26, 2021
Author

georgemccabe Aug 26, 2021
Maintainer

hsoh-u
Aug 25, 2021
Collaborator