Merge pull request #7 from gantian127/release-0.2.0

Release 0.2.0

.github/workflows/test.yml

@@ -20,7 +20,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
 
     steps:
       - uses: actions/checkout@v4
@@ -29,17 +29,9 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
-      - name: API Key File
-        shell: bash
+      - name: Test
         env:
           CDS_API_KEY: ${{secrets.CDS_API_KEY}}
-        run: |
-          echo "url: https://cds.climate.copernicus.eu/api/v2" >> .cdsapirc
-          echo key: $CDS_API_KEY >> .cdsapirc
-          cat .cdsapirc
-          mv .cdsapirc ~/
-
-      - name: Test
         run: |
           pip install nox
           pip install --upgrade setuptools

README.md

@@ -3,20 +3,21 @@
 [![Documentation Status](https://readthedocs.org/projects/bmi_era5/badge/?version=latest)](https://bmi-era5.readthedocs.io/en/latest/?badge=latest)
 [![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/gantian127/bmi_era5/blob/master/LICENSE.txt)
 
-**Please be aware that the CDS platform was upgraded in September 2024, which has resulted in the current bmi_era5 being incompatible with the latest version of the CDS platform. 
-A new release of bmi_era5 is expected later this year.** 
+**Please note: Starting with release v0.2.0, the New CDS platform is now supported.**
 
 bmi_era5 package is an implementation of the Basic Model Interface ([BMI](https://bmi-spec.readthedocs.io/en/latest/))
 for the [ERA5](https://confluence.ecmwf.int/display/CKB/ERA5) dataset.
-This package uses the [CDS API](https://cds.climate.copernicus.eu/api-how-to) to download the ERA5 dataset and wraps the dataset with BMI for data control and query
-(currently support 3 dimensional ERA5 dataset).
+This package uses the [CDS API](https://cds.climate.copernicus.eu/how-to-api) to download
+the ERA5 dataset and wraps the dataset with BMI for data control and query.
+It currently supports 3-dimensional ERA5
+datasets defined with dimensions such as valid_time (or date), latitude, and longitude.
 
 This package is not implemented for people to use but is the key element to convert the ERA5 dataset into
 a data component ([pymt_era5](https://pymt-era5.readthedocs.io/)) for
 the [PyMT](https://pymt.readthedocs.io/en/latest/?badge=latest) modeling framework developed
 by Community Surface Dynamics Modeling System ([CSDMS](https://csdms.colorado.edu/wiki/Main_Page)).
 
-If you have any suggestion to improve the current function, please create a github issue
+If you have any suggestion to improve the current function, please create a GitHub issue
 [here](https://github.com/gantian127/bmi_era5/issues).
 
 
@@ -77,7 +78,7 @@ c.retrieve(
 dataset = xarray.open_dataset("download.nc")
 
 # select 2 meter temperature on 2021-01-01 at 00:00
-air_temp = dataset.t2m.isel(time=0)
+air_temp = dataset.t2m.isel(valid_time=0)
 
 # plot data
 air_temp.plot(figsize=(9, 5))

docs/source/index.md

@@ -8,7 +8,8 @@
 [bmi_era5 package][bmi_era5-github] is an implementation of
 the [Basic Model Interface (BMI)][bmi-docs] for the [ERA5][ERA5] dataset.
 This package uses the [CDS API][cds-api] to download the ERA5 dataset and wraps the
-dataset with BMI for data control and query (currently support 3 dimensional ERA5 dataset).
+dataset with BMI for data control and query. It currently supports 3-dimensional ERA5
+datasets defined with dimensions such as valid_time (or date), latitude, and longitude.
 
 This package is not implemented for people to use but is the key element to convert the ERA5 dataset into
 a data component ([pymt_era5][pymt_era5]) for the [PyMT][pymt-docs]
@@ -80,7 +81,7 @@ c.retrieve(
 dataset = xarray.open_dataset("download.nc")
 
 # select 2 metre temperature on 2021-01-01 at 00:00
-air_temp = dataset.t2m.isel(time=0)
+air_temp = dataset.t2m.isel(valid_time=0)
 
 # plot data
 air_temp.plot(figsize=(9, 5))
@@ -179,7 +180,7 @@ data_comp.finalize()
 [bmi-docs]: https://bmi.readthedocs.io
 [csdms]: https://csdms.colorado.edu
 [pymt-docs]: https://pymt.readthedocs.io
-[cds-api]: https://cds.climate.copernicus.eu/api-how-to
+[cds-api]: https://cds.climate.copernicus.eu/how-to-api
 [bmi_era5-github]: https://github.com/gantian127/bmi_era5/
 [ERA5]: https://confluence.ecmwf.int/display/CKB/ERA5
 [bmi_era5-notebook]: https://github.com/gantian127/bmi_era5/blob/master/notebooks/bmi_era5.ipynb

notebooks/bmi_era5.ipynb

@@ -49,7 +49,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "bmi_era5 package is an implementation of the Basic Model Interface ([BMI](https://bmi.readthedocs.io/en/latest/)) for the [ERA5](https://confluence.ecmwf.int/display/CKB/ERA5) dataset. This package uses the [CDS API](https://cds.climate.copernicus.eu/api-how-to) to download the ERA5 dataset and wraps the dataset with BMI for data control and query (currently support 3 dimensional ERA5 dataset). This package is not implemented for people to use but is the key element to convert the ERA5 dataset into a data component for the [PyMT](https://pymt.readthedocs.io/en/latest/?badge=latest) modeling framework developed by Community Surface Dynamics Modeling System ([CSDMS](https://csdms.colorado.edu/wiki/Main_Page)).\n",
+    "bmi_era5 package is an implementation of the Basic Model Interface ([BMI](https://bmi.readthedocs.io/en/latest/)) for the [ERA5](https://confluence.ecmwf.int/display/CKB/ERA5) dataset. This package uses the [CDS API](https://cds.climate.copernicus.eu/how-to-api) to download the ERA5 dataset and wraps the dataset with BMI for data control and query (currently support 3-dimensional ERA5 dataset). This package is not implemented for people to use but is the key element to convert the ERA5 dataset into a data component for the [PyMT](https://pymt.readthedocs.io/en/latest/?badge=latest) modeling framework developed by Community Surface Dynamics Modeling System ([CSDMS](https://csdms.colorado.edu/wiki/Main_Page)).\n",
     "\n",
     "   \n",
     "To install bmi_era5 package, please follow the instructions [here](https://github.com/gantian127/bmi_era5/#install-package)."
@@ -59,7 +59,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can uncomment the code below to use install_cds( ) to install the CDS API Key file. This file is required for data download. So please make sure you have already created an account at the [CDS registration page](https://cds.climate.copernicus.eu/#!/home) and obtained your CDS API Key. For more details please check [here](https://cds.climate.copernicus.eu/api-how-to)."
+    "You can uncomment the code below to use install_cds( ) to install the CDS API Key file. This file is required for data download. So please make sure you have already created an account at the [CDS registration page](https://cds.climate.copernicus.eu/#!/home) and obtained your CDS API Key. For more details please check [here](https://cds.climate.copernicus.eu/how-to-api)."
    ]
   },
   {
@@ -152,7 +152,7 @@
     "dataset = xarray.open_dataset(\"download.nc\")\n",
     "\n",
     "# select 2 metre temperature on 2021-01-01 at 00:00\n",
-    "air_temp = dataset.t2m.isel(time=0)"
+    "air_temp = dataset.t2m.isel(valid_time=0)"
    ]
   },
   {
@@ -392,7 +392,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.7"
+   "version": "3.13.0"
   }
  },
  "nbformat": 4,

notebooks/installation.py

@@ -5,7 +5,7 @@
 
 def install_cds(config_path=""):
     # create configuration file
-    url = "https://cds.climate.copernicus.eu/api/v2"
+    url = "https://cds.climate.copernicus.eu/api"
     key = input("Enter Your CDS API Key: ")
     config_content = f"url: {url} \nkey: {key}"
 
@@ -16,8 +16,3 @@ def install_cds(config_path=""):
         config_file.write(config_content)
 
     print("CDS API Key file is created.")
-
-    # # install cdsapi
-    # install = subprocess.run(["pip", "install", "cdsapi"], check=True)
-    # if install.returncode == 0:
-    #     print('cdsapi package is successfully installed.')

noxfile.py

@@ -15,6 +15,15 @@ def test(session: nox.Session) -> None:
     """Run the tests."""
     session.install(".[testing]")
 
+    # create API key file for ERA5 (need GitHub secret)
+    url = "https://cds.climate.copernicus.eu/api"
+    key = os.environ.get("CDS_API_KEY")
+    config_content = f"url: {url} \nkey: {key}"
+    home_dir = os.path.expanduser("~")
+    config_path = os.path.join(home_dir, ".cdsapirc")
+    with open(config_path, "w") as config_file:
+        config_file.write(config_content)
+
     args = ["--cov", PROJECT, "-vvv"] + session.posargs
 
     if "CI" in os.environ:

pyproject.toml

@@ -22,6 +22,7 @@ classifiers = [
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
     "Programming Language :: Python :: Implementation :: CPython",
     "Topic :: Scientific/Engineering :: Hydrology",
     "Topic :: Scientific/Engineering :: Physics",
@@ -33,7 +34,8 @@ dependencies = [
     "pyyaml",
     "requests",
     "xarray",
-    "cdsapi",
+    "cdsapi >= 0.7.2",
+    "cftime",
 ]
 dynamic = [
     "version",

requirements.txt

@@ -1,4 +1,6 @@
 bmipy
+cdsapi>=0.7.2
+cftime
 netcdf4
 numpy
 pyyaml

src/bmi_era5/_version.py

@@ -1,3 +1,3 @@
 from __future__ import annotations
 
-__version__ = "0.1.4"
+__version__ = "0.2.0"

src/bmi_era5/bmi.py

@@ -462,14 +462,8 @@ def get_value_ptr(self, name: str) -> numpy.ndarray:
         """
         # return a reference of all the value at current time step.
         # mainly for input data. not useful for scalar value
-        add_offset = self._dataset[self._var_name_mapping[name]].add_offset
-        scale_factor = self._dataset[self._var_name_mapping[name]].scale_factor
-
-        return (
-            self._dataset[self._var_name_mapping[name]].values[self._time_index]
-            * scale_factor
-            + add_offset
-        )
+
+        return self._dataset[self._var_name_mapping[name]].values[self._time_index]
 
     def get_var_grid(self, name: str) -> int:
         """Get grid identifier for the given variable.

src/bmi_era5/utils.py

@@ -1,8 +1,11 @@
 from __future__ import annotations
 
 import os.path
+from datetime import datetime
 
 import cdsapi
+import cftime
+import numpy as np
 import xarray as xr
 
 
@@ -81,17 +84,44 @@ def get_time_info(self):
 
         # time values are float in BMI time function
         if self._data:
-            time_info = {
-                "start_time": float(self._data.time.values[0]),
-                "time_step": 0.0
-                if len(self._data.time.values) == 1
-                else float(self._data.time.values[1] - self._data.time.values[0]),
-                "end_time": float(self._data.time.values[-1]),
-                "total_steps": len(self._data.time.values),
-                "time_units": self._data.time.units,
-                "calendar": self._data.time.calendar,
-                "time_value": self._data.time.values.astype("float"),
-            }
+            if "valid_time" in self._data.keys():
+                time_info = {
+                    "start_time": float(self._data.valid_time.values[0]),
+                    "time_step": 0.0
+                    if len(self._data.valid_time.values) == 1
+                    else float(
+                        self._data.valid_time.values[1]
+                        - self._data.valid_time.values[0]
+                    ),
+                    "end_time": float(self._data.valid_time.values[-1]),
+                    "total_steps": len(self._data.valid_time.values),
+                    "time_units": self._data.valid_time.units,
+                    "calendar": self._data.valid_time.calendar,
+                    "time_value": self._data.valid_time.values.astype("float"),
+                }
+            elif "date" in self._data.keys():
+                # convert date time to CF convention values
+                date_objs = [
+                    datetime.strptime(str(date_value), "%Y%m%d")
+                    for date_value in self._data.date.values
+                ]
+                time_units = "seconds since 1970-01-01"
+                calendar = "proleptic_gregorian"
+                cf_dates = cftime.date2num(
+                    date_objs, units=time_units, calendar=calendar
+                )
+
+                time_info = {
+                    "start_time": float(cf_dates[0]),
+                    "time_step": 0.0
+                    if len(cf_dates) == 1
+                    else float(cf_dates[1] - cf_dates[0]),
+                    "end_time": float(cf_dates[-1]),
+                    "total_steps": len(cf_dates),
+                    "time_units": time_units,
+                    "calendar": calendar,
+                    "time_value": np.array(cf_dates, dtype=float),
+                }
 
         return time_info
 
@@ -101,16 +131,16 @@ def get_var_info(self):
         if self._data:
             for var_name in self._data.data_vars:
                 var = self._data.data_vars[var_name]
-
-                var_info[var.long_name] = {
-                    "var_name": var_name,
-                    "dtype": type(var.scale_factor).__name__
-                    if "scale_factor" in var.attrs.keys()
-                    else str(var.dtype),
-                    "itemsize": var.values.itemsize,
-                    "nbytes": var.values[0].nbytes,  # current time step nbytes
-                    "units": var.units,
-                    "location": "node",
-                }
+                if var.ndim >= 3:
+                    var_info[var.long_name] = {
+                        "var_name": var_name,
+                        "dtype": type(var.scale_factor).__name__
+                        if "scale_factor" in var.attrs.keys()
+                        else str(var.dtype),
+                        "itemsize": var.values.itemsize,
+                        "nbytes": var.values[0].nbytes,  # current time step nbytes
+                        "units": var.units,
+                        "location": "node",
+                    }
 
         return var_info

tests/util_test.py

@@ -24,6 +24,23 @@
     )
 ]
 
+parameters2 = [
+    (
+        "reanalysis-era5-single-levels-monthly-means",
+        "monthly_mean.nc",
+        {
+            "product_type": ["monthly_averaged_reanalysis"],
+            "variable": ["2m_dewpoint_temperature"],
+            "year": ["2022"],
+            "month": ["01", "02", "03", "04"],
+            "time": ["00:00"],
+            "data_format": "netcdf",
+            "download_format": "unarchived",
+            "area": [39, -106, 36, -103],
+        },
+    )
+]
+
 
 @pytest.mark.parametrize("name, file, era5_req", parameters)
 def test_get_data(tmpdir, name, file, era5_req):
@@ -62,21 +79,22 @@ def test_get_var_info(tmpdir, name, file, era5_req):
 
     era5.get_data(name, era5_req, path)
     var_info_2 = era5.get_var_info()
-    assert len(var_info_2) == 2
+
     assert "Total precipitation" in var_info_2.keys()
     assert "2 metre temperature" in var_info_2.keys()
 
     var = var_info_2["Total precipitation"]
     assert var["var_name"] == "tp"
-    assert var["dtype"] == "float64"
-    assert var["itemsize"] == 2
-    assert var["nbytes"] == 1218
+    assert var["dtype"] == "float32"
+    assert var["itemsize"] == 4
+    assert var["nbytes"] == 2436
     assert var["units"] == "m"
     assert var["location"] == "node"
 
 
 @pytest.mark.parametrize("name, file, era5_req", parameters)
-def test_get_time_info(tmpdir, name, file, era5_req):
+def test_get_time_info_valid_time(tmpdir, name, file, era5_req):
+    """Test when time variable is valid_time"""
     path = os.path.join(tmpdir, file)
 
     era5 = Era5Data()
@@ -87,9 +105,31 @@ def test_get_time_info(tmpdir, name, file, era5_req):
     era5.get_data(name, era5_req, path)
     time_info_2 = era5.get_time_info()
 
-    assert time_info_2["start_time"] == 1060680
-    assert time_info_2["end_time"] == 1060682
-    assert time_info_2["time_step"] == 1
+    assert time_info_2["start_time"] == 1609459200.0
+    assert time_info_2["end_time"] == 1609466400.0
+    assert time_info_2["time_step"] == 3600.0
     assert time_info_2["total_steps"] == 3
-    assert time_info_2["time_units"] == "hours since 1900-01-01 00:00:00.0"
-    assert time_info_2["calendar"] == "gregorian"
+    assert time_info_2["time_units"] == "seconds since 1970-01-01"
+    assert time_info_2["calendar"] == "proleptic_gregorian"
+
+
+@pytest.mark.parametrize("name, file, era5_req", parameters2)
+def test_get_time_info_date(tmpdir, name, file, era5_req):
+    """Test when time variable is date"""
+
+    path = os.path.join(tmpdir, file)
+
+    era5 = Era5Data()
+    time_info_1 = era5.get_time_info()
+
+    assert time_info_1 == {}
+
+    era5.get_data(name, era5_req, path)
+    time_info_2 = era5.get_time_info()
+
+    assert time_info_2["start_time"] == 1640995200.0
+    assert time_info_2["end_time"] == 1648771200.0
+    assert time_info_2["time_step"] == 2678400.0
+    assert time_info_2["total_steps"] == 4
+    assert time_info_2["time_units"] == "seconds since 1970-01-01"
+    assert time_info_2["calendar"] == "proleptic_gregorian"