Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate replacing pygrib with cfgrib(probably with xarray) for the ingest processes #221

Closed
randytpierce opened this issue Jul 19, 2023 · 2 comments · Fixed by #230
Closed
Assignees
Labels
task Tasks break a project down into discrete steps VXingest issues related to the VXingest project

Comments

@randytpierce
Copy link
Contributor

Describe what needs to be done here.
The Pygrib python package has been having build and install problems so we need to find out if cfgrib is better supported and works as well. This is a blocker for the ingest work.

@randytpierce randytpierce added VXingest issues related to the VXingest project task Tasks break a project down into discrete steps labels Jul 19, 2023
@randytpierce randytpierce self-assigned this Jul 19, 2023
@randytpierce
Copy link
Contributor Author

Reference repo https://github.com/ecmwf/cfgrib. This is very possible and now work is proceeding on making this happen per our discussion in the vxIngest meeting July 25th.
Here are a few notes for processing a test model output file with xarray engine='cfgrib'...
python
import xarray
import cfgrib

# how to get surface variables (level 2 meter typeOfLevel heightAboveGround)...
# ds2 = xr.open_dataset('/opt/data/grib2_to_cb/input_files/2125214000000',  engine='cfgrib',backend_kwargs={'filter_by_keys':{'typeOfLevel': 'heightAboveGround', 'stepType':'instant', 'level':2}})
# >>> sorted(ds2.variables  ['d2m', 'heightAboveGround', 'latitude', 'longitude', 'pt', 'r2', 'sh2', 'step', 't2m', 'time', 'valid_time']
# ds2.variables['r2']
# ds2.variables['r2'].values
# How to get wind data at 10 m
# >>> dsh = xr.open_dataset('/opt/data/grib2_to_cb/input_files/2125214000000',  engine='cfgrib',backend_kwargs={'filter_by_keys':{'typeOfLevel': 'heightAboveGround', 'stepType':'instant', 'level':10}})
# how to get ceiling data
# >>> dsc = xr.open_dataset('/opt/data/grib2_to_cb/input_files/2125214000000',  engine='cfgrib',backend_kwargs={'filter_by_keys':{'typeOfLevel': 'cloudCeiling', 'stepType':'instant'}})
# sorted(dsc.variables)
# dsc.variables['cloudCeiling'].dims
# dsc.variables['cloudCeiling'].data
# dsc.coords['latitude']
# dsc.coords['longitude']

@randytpierce
Copy link
Contributor Author

We have demonstrated that we can make this work and have decided as a team to implement cfgrib via xarray so I am closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Tasks break a project down into discrete steps VXingest issues related to the VXingest project
Projects
None yet
1 participant