Drop Python3.8, support Pandas v2.2, reduce Warnings (#1565)

### What kind of change does this PR introduce?

* Drops Python3.8 references and marks Python3.9 as base requirement.
* Drops older `xarray` and `pandas` support.
* Updates documentation and CI for dropped Python3.8.
* Raises required versions of `scipy` and `numpy` (`1.9.0` and `1.20.0`,
respectively).
* Removes the `winter_storm` indice and indicator (deprecated in `xclim`
v0.46.0).
* Adopts the new syntax for frequency codes:
  - "A" is no more
  - Y -> YE, Q -> QE, M -> ME
  - H -> h, T -> min, S -> s, L -> ms, U -> us, N -> ns
* Addresses several `DeprecationWarning` and `RuntimeWarning` messages
related to `numpy`, `xarray`, and `pint`

### Does this PR introduce a breaking change?

**Yes**, Python3.8 is no longer supported and base dependency versions
have been augmented:
- `numpy>=1.20.0`
- `pandas>=2.2.0`
- `scipy>=1.9.0`
- `xarray>=2023.11.0`

### Other information:

~Python3.12 support is still not possible due to missing Python3.12
builds for `numba`. (This is being addressed in #1613)~ - This has been
addressed.

A Pull Request has been opened to address `DeprecationWarning`s stemming
from `numpy` calls in `eofs` (ajdawson/eofs#148;
`eofs` has been removed due to licensing-related issues (#1621))

.github/workflows/main.yml

@@ -41,7 +41,7 @@ jobs:
     strategy:
       matrix:
         python-version:
-          - "3.8"
+          - "3.9"
     steps:
       - name: Harden Runner
         uses: step-security/harden-runner@eb238b55efaa70779f274895e782ed17c84f2895 # v2.6.1
@@ -117,9 +117,6 @@ jobs:
     strategy:
       matrix:
         include:
-          - tox-env: py38-coverage
-            python-version: "3.8"
-            markers: -m 'not slow'
           - tox-env: py39-coverage-sbck
             python-version: "3.9"
             markers: -m 'not slow'

.pre-commit-config.yaml

@@ -6,7 +6,7 @@ repos:
     rev: v3.15.0
     hooks:
       - id: pyupgrade
-        args: ['--py38-plus']
+        args: ['--py39-plus']
         exclude: 'xclim/core/indicator.py'
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.5.0
@@ -59,7 +59,7 @@ repos:
     rev: 1.7.1
     hooks:
       - id: nbqa-pyupgrade
-        args: [ '--py38-plus' ]
+        args: [ '--py39-plus' ]
       - id: nbqa-black
         additional_dependencies: [ 'black==24.1.1' ]
       - id: nbqa-isort

CHANGES.rst

@@ -8,6 +8,7 @@ Contributors to this version: Juliette Lavoie (:user:`juliettelavoie`), Pascal B
 
 Announcements
 ^^^^^^^^^^^^^
+* `xclim` no longer supports Python3.8. (:issue:`1268`, :pull:`1565`).
 * `xclim` now officially supports Python3.12 (requires `numba>=0.59.0`). (:pull:`1613`).
 * `xclim` now adheres to the `Semantic Versioning 2.0.0 <https://semver.org/>`_ specification. (:issue:`1556`, :pull:`1569`).
 * The `xclim` repository now uses `GitHub Discussions <https://github.com/Ouranosinc/xclim/discussions>`_ to offer help for users, coordinate translation efforts, and support general Q&A for the `xclim` community. The `xclim` `Gitter` room has been deprecated in favour of GitHub Discussions. (:issue:`1571`, :pull:`1572`).
@@ -23,11 +24,20 @@ New features and enhancements
 
 Breaking changes
 ^^^^^^^^^^^^^^^^
-* `bump2version` has been replaced with `bump-my-version` to bump the version number using configurations set in the `pyproject.toml` file. (:issue:`1557`, :pull:`1569`).
+* `xclim` base Python version has been raised to `python>=3.9`. Python3.9+ coding conventions are now supported. (:issue:`1268`, :pull:`1565`).
+* `xclim` base dependencies have been raised to `pandas>=2.2.0` and `xarray>=2023.11.0` to reflect changes to time frequency codes introduced in `pandas==2.2.0`. (:issue:`1534`, :pull:`1565`; see also: `pydata/xarray GH/8394 <https://github.com/pydata/xarray/issues/8394>`_ and ). Many default frequency string outputs have been modified (:
+    * 'Y' (year) -> 'YE' (year end). (see: `pandas PR/55792 <https://github.com/pandas-dev/pandas/pull/55792>`_).
+    * 'M' (month) -> 'ME' (month end). (see: `pandas PR/52064 <https://github.com/pandas-dev/pandas/pull/52064>`_).
+    * 'Q' (quarter) -> 'QE' (quarter end). (see: `pandas PR/55553 <https://github.com/pandas-dev/pandas/pull/55553>`_)
+    * 'A' and 'AS' have been removed (use 'YE' and 'YS' instead). (see: `pandas PR/55252 <https://github.com/pandas-dev/pandas/pull/55252>`_). ('YE' is only supported for cftime data in `xarray >= 2024.1.1`).
+    * 'T' (minute), 'L' (millisecond), 'U' (microsecond), and 'N' (nanosecond) -> 'min', 'ms', 'us', and 'ns'. (see: `pandas PR/54061 <https://github.com/pandas-dev/pandas/pull/54061>`_).
+* `bump2version` has been replaced with `bump-my-version` to bump the version number using configurations set in the ``pyproject.toml`` file. (:issue:`1557`, :pull:`1569`).
 * `xclim`'s units registry and units formatting are now extended from `cf-xarray`. The exponent sign "^" is now never added in the ``units`` attribute. For example, square meters are given as "m2" instead of "m^2" by xclim, both are still accepted as input. (:issue:`1010`, :pull:`1590`).
 * `yamale` is now listed as a core dependency (was previously listed in the `dev` installation recipe). (:issue:`1595`, :pull:`1596`).
 * Due to a licensing limitation, the calculation of empirical orthogonal function  based on `eofs` (``xclim.sdba.properties.first_eof``) has been removed from `xclim`. (:issue:`1620`, :pull:`1621`).
 * `black` formatting style has been updated to the 2024 stable conventions. `isort` has been added to the `dev` installation recipe. (:pull:`1626`).
+* The indice and indicator for ``winter_storm`` has been removed (deprecated since `xclim` v0.46.0 in favour of ``snd_storm_days``). (:pull:`1565`).
+* `xclim` has dropped support for `scipy` version below v1.9.0 and `numpy` versions below v1.20.0. (:pull:`1565`).
 
 Bug fixes
 ^^^^^^^^^
@@ -38,19 +48,20 @@ Bug fixes
 
 Internal changes
 ^^^^^^^^^^^^^^^^
-* The `flake8` configuration has been migrated from `setup.cfg` to `.flake8`; `setup.cfg` has been removed. (:pull:`1569`)
-* The `bump-version.yml` workflow has been adjusted to bump the `patch` version when the last version is determined to have been a `release` version; otherwise, the `build` version is bumped. (:issue:`1557`, :pull:`1569`).
+* The `flake8` configuration has been migrated from ``setup.cfg`` to ``.flake8``; ``setup.cfg`` has been removed. (:pull:`1569`)
+* The ``bump-version.yml`` workflow has been adjusted to bump the `patch` version when the last version is determined to have been a `release` version; otherwise, the `build` version is bumped. (:issue:`1557`, :pull:`1569`).
 * The GitHub Workflows now use the `step-security/harden-runner` action to monitor source code, actions, and dependency safety. All workflows now employ more constrained permissions rule sets to prevent security issues. (:pull:`1577`, :pull:`1578`, :pull:`1597`).
-* Updated the CONTRIBUTING.rst directions to showcase the new versioning system. (:issue:`1557`, :pull:`1573`).
-* The `codespell` library is now a development dependency for the `dev` installation recipe with configurations found within `pyproject.toml`. This is also now a linting step and integrated as a `pre-commit` hook. For more information, see the `codespell documentation <https://github.com/codespell-project/codespell>`_ (:pull:`1576`).
+* Updated the ``CONTRIBUTING.rst`` directions to showcase the new versioning system. (:issue:`1557`, :pull:`1573`).
+* The `codespell` library is now a development dependency for the `dev` installation recipe with configurations found within ``pyproject.toml``. This is also now a linting step and integrated as a `pre-commit` hook. For more information, see the `codespell documentation <https://github.com/codespell-project/codespell>`_ (:pull:`1576`).
 * Climate indicators search page now prioritizes the "official" indicators (atmos, land, seaIce and generic), virtual submodules can be added to search through checkbox option. (:issue:`1559`, :pull:`1593`).
 * The OpenSSF StepSecurity bot has contributed some changes to the workflows and pre-commit. (:issue:`1181`, :pull:`1606`):
     * Dependabot has been configured to monitor the `xclim` repository for dependency updates. The ``actions-version-updater.yml`` workflow has been deprecated.
     * GitHub Actions are now pinned to their commit hashes to prevent unexpected changes in the future.
     * A new GitHub Workflow (``workflow-warning.yml``) has been added to warn maintainers when a forked repository has been used to open a Pull Request that modifies GitHub Workflows.
     * `pylint` has been configured to provide some overhead checks of the `xclim` codebase as well as run as part of `xclim`'s `pre-commit` hooks.
     * Some small adjustments to code organization to address `pylint` errors.
-* `dev` formatting tools (`black`, `blackdoc`, `isort`) are now pinned to their `pre-commit` hook version equivalents in both `pyproject.toml` and `tox.ini`. (:pull:`1626`).
+* `dev` formatting tools (`black`, `blackdoc`, `isort`) are now pinned to their `pre-commit` hook version equivalents in both ``pyproject.toml`` and ``tox.ini``. (:pull:`1626`).
+* `black`, `isort`, and `pyupgrade` code formatters no longer target Python3.8 coding style conventions. (:pull:`1565`).
 
 v0.47.0 (2023-12-01)
 --------------------

CONTRIBUTING.rst

@@ -121,8 +121,8 @@ Ready to contribute? Here's how to set up `xclim` for local development.
 
 #. Create a development environment. We recommend using ``conda``::
 
-    $ conda create -n xclim python=3.8 --file=environment.yml
-    $ pip install -e .[dev]
+    $ conda create -n xclim python=3.10 --file=environment.yml
+    $ python -m pip install -e ".[dev]"
 
 #. Create a branch for local development::
 
@@ -162,10 +162,10 @@ Ready to contribute? Here's how to set up `xclim` for local development.
 
    Alternatively, one can use ``$ tox`` to run very specific testing configurations, as GitHub Workflows would do when a Pull Request is submitted and new commits are pushed::
 
-    $ tox -e py38  # run tests on Python 3.8
-    $ tox -e py39-upstream-doctest  # run tests on Python 3.9, including doctests, with upstream dependencies
-    $ tox -e py310 -- -m "not slow  # run tests on Python 3.10, excluding "slow" marked tests
-    $ tox -e py311  # run tests on Python 3.11
+    $ tox -e py39  # run tests on Python 3.9
+    $ tox -e py310-upstream-doctest  # run tests on Python 3.10, including doctests, with upstream dependencies
+    $ tox -e py311 -- -m "not slow  # run tests on Python 3.11, excluding "slow" marked tests
+    $ tox -e py312-numba -- -m "not slow  # run tests on Python 3.12, installing upstream `numba`, excluding "slow" marked tests
     $ tox -e notebooks_doctests  # run tests using the base Python on doctests and evaluate all notebooks
     $ tox -e offline  # run tests using the base Python, excluding tests requiring internet access
 
@@ -243,7 +243,7 @@ Before you submit a pull request, please follow these guidelines:
     If you aren't accustomed to writing documentation in reStructuredText (`.rst`), we encourage you to spend a few minutes going over the
     incredibly well-summarized `reStructuredText Primer`_ from the sphinx-doc maintainer community.
 
-#. The pull request should work for Python 3.8, 3.9, 3.10, and 3.11 as well as raise test coverage.
+#. The pull request should work for Python 3.9, 3.10, 3.11, and 3.12 as well as raise test coverage.
    Pull requests are also checked for documentation build status and for `PEP8`_ compliance.
 
    The build statuses and build errors for pull requests can be found at: https://github.com/Ouranosinc/xclim/actions

docs/explanation.rst

@@ -5,11 +5,6 @@ Why use xclim?
 Purpose
 =======
 
-.. important::
-
-    The content of this section is actively being developed in the forthcoming paper submission to JOSS.
-    This section will be updated and finalized when the wording has been agreed upon in :pull:`250`
-
 `xclim` aims to position itself as a climate services tool for any researchers interested in using Climate and Forecast Conventions (`CF-Conventions <https://cfconventions.org/>`_) compliant datasets to perform climate analyses. This tool is optimized for working with Big Data in the climate science domain and can function as an independent library for one-off analyses in *Jupyter Notebooks* or as a backend engine for performing climate data analyses via **Web Processing Services** (`WPS <https://www.ogc.org/standard/wps/>`_; e.g. `Finch <https://github.com/bird-house/finch>`_). It was primarily developed targeting Earth and Environmental Science audiences and researchers, originally for calculating climate indicators for the Canadian government web service `ClimateData.ca <https://climatedata.ca/>`_.
 
 The primary domains that `xclim` is built for are in calculating climate indicators, performing statistical correction / bias adjustment of climate model output variables or simulations, and in performing climate model simulation ensemble statistics.

environment.yml

@@ -4,27 +4,27 @@ channels:
   - conda-forge
   - defaults
 dependencies:
-  - python >=3.8
+  - python >=3.9
   - astroid
   - boltons >=20.1
   - bottleneck >=1.3.1
   - cf_xarray >=0.6.1
   - cftime >=1.4.1
   - Click >=8.1
   - dask >=2.6.0
-  - importlib-resources  # For Python3.8
   - jsonpickle
   - lmoments3
   - numba
-  - numpy >=1.16
-  - pandas >=0.23,<2.2
+  - numpy >=1.20.0
+  - pandas >=2.2.0
   - pint >=0.9
   - poppler >=0.67
+  - pyarrow  # Strongly encouraged for Pandas v2.2.0+
   - pyyaml
   - scikit-learn >=0.21.3
-  - scipy >=1.2
+  - scipy >=1.9.0
   - statsmodels
-  - xarray >=2022.06.0,<2023.11.0
+  - xarray >=2023.11.0
   - yamale
   # Extras
   - flox

pyproject.toml

@@ -12,7 +12,7 @@ maintainers = [
   {name = "Pascal Bourgault", email = "[email protected]"}
 ]
 readme = {file = "README.rst", content-type = "text/x-rst"}
-requires-python = ">=3.8.0"
+requires-python = ">=3.9.0"
 keywords = ["xclim", "xarray", "climate", "climatology", "bias correction", "ensemble", "indicators", "analysis"]
 license = {file = "LICENSE"}
 classifiers = [
@@ -23,7 +23,6 @@ classifiers = [
   "Natural Language :: English",
   "Operating System :: OS Independent",
   "Programming Language :: Python :: 3",
-  "Programming Language :: Python :: 3.8",
   "Programming Language :: Python :: 3.9",
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
@@ -35,24 +34,22 @@ dependencies = [
   "boltons>=20.1",
   "bottleneck>=1.3.1",
   # cf-xarray is differently named on conda-forge
-  "cf-xarray>=0.6.1,<0.8.5; python_version == '3.8'",
-  "cf-xarray>=0.6.1; python_version >= '3.9'",
+  "cf-xarray>=0.6.1",
   "cftime>=1.4.1",
   "Click>=8.1",
   "dask[array]>=2.6",
-  "importlib-resources; python_version == '3.8'",
   "jsonpickle",
   "lmoments3>=1.0.5",
   "numba",
-  "numpy>=1.16",
-  "pandas>=0.23,<2.0; python_version == '3.8'",
-  "pandas>=0.23,<2.2; python_version >= '3.9'",
+  "numpy>=1.20.0",
+  "pandas>=2.2",
   "pint>=0.10",
+  "pyarrow", # Strongly encouraged for pandas v2.2.0+
   "pyyaml",
   "scikit-learn>=0.21.3",
-  "scipy>=1.2",
+  "scipy>=1.9.0",
   "statsmodels",
-  "xarray>=2022.06.0,<2023.11.0",
+  "xarray>=2023.11.0",
   "yamale"
 ]
 
@@ -71,6 +68,7 @@ dev = [
   "ipython",
   "isort ==5.13.2",
   "mypy",
+  "nbconvert",
   "nbqa",
   "nbval",
   "netCDF4 >=1.4",
@@ -119,10 +117,10 @@ xclim = "xclim.cli:cli"
 
 [tool.black]
 target-version = [
-  "py38",
   "py39",
   "py310",
-  "py311"
+  "py311",
+  "py312"
 ]
 
 [tool.bumpversion]
@@ -199,12 +197,12 @@ exclude = [
 
 [tool.isort]
 profile = "black"
-py_version = 38
+py_version = 39
 append_only = true
 add_imports = "from __future__ import annotations"
 
 [tool.mypy]
-python_version = 3.8
+python_version = 3.9
 show_error_codes = true
 warn_return_any = true
 warn_unused_configs = true
@@ -252,13 +250,12 @@ markers = [
 [tool.ruff]
 src = ["xclim"]
 line-length = 150
-target-version = "py38"
+target-version = "py39"
 exclude = [
   ".git",
   "docs",
   "build",
-  ".eggs",
-  "tests"
+  ".eggs"
 ]
 ignore = [
   "D205",

tests/conftest.py

@@ -141,7 +141,7 @@ def prsnd_series():
 def pr_hr_series():
     """Return precipitation hourly time series."""
     _pr_hr_series = partial(
-        test_timeseries, start="1/1/2000", variable="pr", units="kg m-2 s-1", freq="1H"
+        test_timeseries, start="1/1/2000", variable="pr", units="kg m-2 s-1", freq="h"
     )
     return _pr_hr_series
 

tests/test_analog.py

@@ -5,8 +5,6 @@
 import pandas as pd
 import pytest
 from numpy.testing import assert_almost_equal
-from packaging.version import Version
-from scipy import __version__ as __scipy_version__
 from scipy import integrate, stats
 from sklearn import datasets
 
@@ -60,11 +58,6 @@ def test_exact_randn(exact_randn):
 @pytest.mark.slow
 @pytest.mark.parametrize("method", xca.metrics.keys())
 def test_spatial_analogs(method, open_dataset):
-    if method in ["nearest_neighbor", "kldiv"] and Version(__scipy_version__) < Version(
-        "1.6.0"
-    ):
-        pytest.skip("Method not supported in scipy<1.6.0")
-
     diss = open_dataset("SpatialAnalogs/dissimilarity")
     data = open_dataset("SpatialAnalogs/indicators")
 
@@ -138,10 +131,6 @@ def test_compare_with_matlab(self):
         assert_almost_equal(dm, 2.8463, 4)
 
 
-@pytest.mark.skipif(
-    Version(__scipy_version__) < Version("1.6.0"),
-    reason="Not supported in scipy<1.6.0",
-)
 class TestNN:
     def test_simple(self, random):
         d = 2
@@ -244,10 +233,6 @@ def func(x):
 
 
 @pytest.mark.slow
-@pytest.mark.skipif(
-    Version(__scipy_version__) < Version("1.6.0"),
-    reason="Not supported in scipy<1.6.0",
-)
 class TestKLDIV:
     #
     def test_against_analytic(self, random):

tests/test_atmos.py

@@ -272,7 +272,7 @@ def test_wind_power_potential_from_3h_series():
     from xclim.testing.helpers import test_timeseries
 
     w = test_timeseries(
-        np.ones(96) * 15, variable="sfcWind", start="7/1/2000", units="m s-1", freq="3H"
+        np.ones(96) * 15, variable="sfcWind", start="7/1/2000", units="m s-1", freq="3h"
     )
     out = atmos.wind_power_potential(wind_speed=w)
 

tests/test_bootstrapping.py

@@ -26,9 +26,9 @@ class Test_bootstrap:
         "var,p,index,freq, cftime",
         (
             ["tas", 98, tg90p, "MS", False],
-            ["tasmin", 98, tn90p, "A-JUL", False],
-            ["tasmax", 98, tx90p, "Q-APR", False],
-            ["tasmax", 98, tx90p, "Q-APR", True],
+            ["tasmin", 98, tn90p, "YS-JUL", False],
+            ["tasmax", 98, tx90p, "QS-APR", False],
+            ["tasmax", 98, tx90p, "QS-APR", True],
             ["tasmin", 2, tn10p, "MS", False],
             ["tasmax", 2, tx10p, "YS", False],
             ["tasmax", 2, tx10p, "YS", True],