GribMagic - generic weather forecast downloader

Simulations are believed by no one except those who conducted them.

Experimental results are believed by everyone except those who conducted them.

ANONYMOUS

About

The goal of the GribMagic project is to unify the download process of public GRIB1/GRIB2 and netCDF data from numerical weather prediction models originating from different organizations and data providers.

This is a work in progress, as such GribMagic is currently considered to be beta software. As this is an early-stage project, contributions are highly appreciated.

Details

The software package includes different code bases. Both provide their features by means of a gribmagic subcommand.

• GribMagic Unity program: gribmagic unity --help
• DWD GRIB Downloader program: gribmagic dwd --help

You can find corresponding invocation examples later in this document.

Setup

Install the ecCodes package by ECMWF.

# Debian Linux
apt-get install --yes libeccodes0 cdo

# macOS/Homebrew
brew install eccodes cdo

Install GribMagic Python package.

pip install gribmagic --upgrade

Run GribMagic Unity program

Ad hoc usage

# List labels of available models.
gribmagic unity list

# Acquire data.
gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z --target=.gribmagic-data

Configuration

To optionally adjust configuration settings, you can define the following environment variables:

GM_DATA_PATH={PATH_TO_PROJECT}/data

GM_MODEL_CONFIG={PATH_TO_PROJECT}/config/model_config.yml"
GM_MODEL_VARIABLES_MAPPING={PATH_TO_PROJECT}/config/model_variables_mapping.yml"
GM_MODEL_VARIABLES_LEVELS_MAPPING={PATH_TO_PROJECT}/config/model_variables_levels_mapping.yml"

ECCODES_DEFINITION_PATH=/usr/share/eccodes/definitions:/usr/local/opt/eccodes/share/eccodes/definitions

Run DWD GRIB Downloader program

# Install DWD GRIB Downloader program.
gribmagic install dwd-grib-downloader

# Acquire wind-specific parameters from ICON-D2.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_d2_wind.py
gribmagic dwd acquire --recipe=recipe_d2_wind.py --timestamp="2021101800" --output=.gribmagic-data/raw

# Acquire assorted parameters from ICON-GLOBAL.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_global_assorted.py
gribmagic dwd acquire --recipe=recipe_global_assorted.py --timestamp="2021101800" --output=.gribmagic-data/raw

When omitting the --timestamp parameter, the most recent modelrun is automatically selected. When omitting the --output parameter, it can be supplied using the GM_DATA_PATH environment variable.

In this manner, the most compact form to invoke gribmagic dwd would be something like:

export GM_DATA_PATH=.gribmagic-data/raw
gribmagic dwd acquire --recipe=recipe_d2_wind.py

Run bbox tool

Extract area of interest from GRIB files using a bounding box.

Extract subset by coordinates, a space-separated list of lat_min lat_max lon_min lon_max.

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid/icon-d2 \
  --bbox=46.0 47.5 14.5 16.8

Extract subset by 2-letter country name.

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid/icon-d2 \
  --country=AT

For plotting the result on a map, install Magics:

# Linux
apt-get install --yes libmagplus3v5

# macOS
make magics-install
make magics-info
export MAGPLUS_HOME=/usr/local/opt/magics-4.9.3

# Install GribMagic with Magics bindings
pip install gribmagic[plotting]

and use the --plot option:

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid \
  --country=AT \
  --plot

Run regridding tool

Acquire grid information files.

gribmagic install dwd-grid-information

Process regridding on a single file. The default is to assume ICON GLOBAL files and the regridding will take place with 0.250° resolution.

gribmagic smith regrid \
  ".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
  --output=.gribmagic-data/regridded/icon-global

When aiming to regrid with 0.125° resolution, please specify the --resolution=0.125 argument, like:

gribmagic smith regrid \
  ".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
  --output=.gribmagic-data/regridded/icon-global \
  --resolution=0.125

Run program in Docker

To use GribMagic in a Docker container, you have to build the Docker image like

docker build --tag gribmagic .

and then invoke it like

docker run -it --volume=$PWD/.gribmagic-data:/var/spool/gribmagic gribmagic:latest gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z

Development

Acquire source code

git clone https://github.com/earthobservations/gribmagic
cd gribmagic

Run software tests

# Run all tests.
make test

# All tests, with code coverage report.
make test-coverage

# Run specific tests, with coverage report.
.venv/bin/pytest -vvv --cov-report=term-missing --cov=gribmagic.smith.regrid -k test_regrid

# Run tests in parallel.
make test-parallel
make test-coverage-parallel

---

Content attributions

The copyright of data sets, images and pictograms are held by their respective owners, unless otherwise noted.

Banner image

• 2 metre temperature from ECMWF, rendered using Magics.
  • Source: MARS labelling or ensemble forecast data (2013-03-25)
  • URL: http://download.ecmwf.int/test-data/magics/2m_temperature.grib