-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vortex is importing some precip grids but not others #104
Comments
@cen4abc if you post sample files I can take a look. Despite communal attempts at conventionality (climate and forecasting metadata conventions) no two data distributors ever format their files in the exact same way which means every file from a new source is an adventure! |
@tombrauer that would be great, thank you. I have placed some sample files here: https://leeds365-my.sharepoint.com/:f:/g/personal/cen4abc_leeds_ac_uk/EhvnQ56Y3vdLtKOTR64rbwQB3ujM44kP799qrQyCnXsFGg?e=1sch7V It might be better to start with the ECWMF files as these are more widely used / available than the NCAS data. Thanks again! |
I investigated the ECMWF file and it is hitting |
Thank you for getting to the botton of this. If you could look at the NCAS dataset at some point as well, that would be much appreciated. |
@cen4abc I'm looking at the NCAS dataset and it does not appear to follow any convention. Vortex is expecting CF. Passing this dataset to Vortex is kind of like passing a Canadian bill to an ATM programed for U.S. legal tender. Does the originator of the dataset have an explanation for what follow variables mean: |
@tombrauer let me talk to NCAS and get back to you. I can guess but not with certainty. |
Hello again, All variables are defined as follows (crr intensity is the one we are interested in): crr - Convective Rainfall Rate Class crr_accum- Convective Hourly Rainfall Accumulation crr_conditions- Common geophysical and processing conditions crr_intensity- convective precipitation rate crr_intensity_pal- RGB palette for crr_intensity crr_pal- RGB palette for crr crr_quality- Common Quality Indicators crr_status_flag- Information on specific NWC GEO CRR processing |
@cen4abc I just published a build that should read in the NCAS dataset: The originators of this dataset didn't do anyone any favors by following no particular convention with the dataset. The dataset uses an irregular grid (kind of like a spider web) that must be re-indexed into a cartesian grid. It takes some time to re-index the 1.8 million grid cells but the good news is that if you are importing multiple grids at a time, the re-indexing only has to be done once for the multiple grids. The dataset does not include a time dimension so there is no conventional way to attribute time. I found a global attribute "nominal_product_time" that gives some notion of time. Inferring time from attributes isn't conventional but I added handling logic to read the attribute if it exists and apply it as both start and end time for the grid. The variable names "crr_accum" and "NWC GEO CRR Convective Hourly Rainfall Accumulation" are non-standard but I added logic to recognize them as precipitation. Standard name table: https://cfconventions.org/Data/cf-standard-names/current/build/cf-standard-name-table.html Because there is only a single time attribute to each grid record, you'll have to use the time-shifter to shift either the start date or the end date of the data to create period cumulative precipitation data. Based on the 3 grids provided I'm guessing you'll want to shift the start time by -15 minutes. Feel free to provide this feedback to the originators of the data. I will quickly run out of time if I add handling logic like this for every non-conventional dataset that's out there. The burden should really be on the originator to clean up their data and make it more transferrable to others. |
@tombrauer - very appreciative of this, thanks. I will raise this with the dataset provider! |
@tombrauer I have just sent you a PM about this. |
Hello,
I have been successfully using vortex (version 0.11.10-rc.1) to import ECWMF ERA5 reanalysis precip data (in both netcdf and grib format) and applying it in HECRAS rain on grid models with some nice results (big thanks to the vortex developers). I have also successfully used MRMS QPE data. However, vortex does not seem to want to import some other precip datasets that I would like to use in HECRAS, namely:
Vortex is either creating an empty output file or gets stuck during the import process. I am quite new to gridded climate datasets and cannot see anything that might be causing vortex an issue. Do you have any ideas as to why vortex is not handling these two datasets, and how you might process them to make them compatible? Below are screenshots of the parameters that are visible when first loading the datasets into vortex.
Many thanks,
Andy
ps. for info, this is the ERA5 reanalysis data that vortex will work with:
The text was updated successfully, but these errors were encountered: