Numpydoc Compliance (#1988)

### What kind of change does this PR introduce?

* Integrates the `numpydoc` library for docstring validation.
* Fixes the `codespell` configuration (previously was not parsing all
relevant files).
* Drops Python3.9 coding conventions everywhere.
* Reformats existing docstrings using `\*` to use `*` instead.
* On a related note, disabled `flake8`'s `RST210` and `RST213` checks
that were flagging `*` symbols in docstrings as unterminated *emphasis*
and **strong** tags.
* Adds a step to the `Indictor.parse_indice` that sanitizes the `*`
symbols from `Parameters` entries.
* Painstakingly modifies all docstrings to be closer to the `numpydoc`
specification.

### Does this PR introduce a breaking change?

It should not. Call signatures and docstrings will be slightly changed.

### Other information:

https://numpydoc.readthedocs.io/en/latest/validation.html

Except for the custom fields that we use to populate our documentation,
the docstrings of `xclim` are very consistent and virtually compliant
with the standard now.

Both `xclim.sdba` and `xclim.core.missing` are not evaluated by
`numpydoc` due to major refactoring efforts that are underway
(https://github.com/Ouranosinc/xsdba) or needed (#2000).

.flake8

@@ -10,6 +10,8 @@ ignore =
 	D,
 	E,
 	F,
+	RST210,
+	RST213,
 	W503
 per-file-ignores =
 	xclim/core/locales.py:RST399

.pre-commit-config.yaml

@@ -6,7 +6,7 @@ repos:
     rev: v3.19.0
     hooks:
       - id: pyupgrade
-        args: ['--py39-plus']
+        args: ['--py310-plus']
         exclude: 'xclim/core/indicator.py'
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v5.0.0
@@ -60,16 +60,16 @@ repos:
     rev: 1.9.1
     hooks:
       - id: nbqa-pyupgrade
-        args: [ '--py39-plus' ]
-        additional_dependencies: [ 'pyupgrade==3.18.0' ]
+        args: [ '--py310-plus' ]
+        additional_dependencies: [ 'pyupgrade==3.19.0' ]
       - id: nbqa-black
         additional_dependencies: [ 'black==24.10.0' ]
   - repo: https://github.com/kynan/nbstripout
     rev: 0.8.1
     hooks:
       - id: nbstripout
         files: '.ipynb'
-        args: [ '--extra-keys', 'metadata.kernelspec' ]
+        args: [ '--extra-keys=metadata.kernelspec' ]
   - repo: https://github.com/pre-commit/pygrep-hooks
     rev: v1.10.0
     hooks:
@@ -98,6 +98,13 @@ repos:
     hooks:
       - id: codespell
         additional_dependencies: [ 'tomli' ]
+        args: [ '--toml=pyproject.toml' ]
+  - repo: https://github.com/numpy/numpydoc
+    rev: v1.8.0
+    hooks:
+      - id: numpydoc-validation
+        # Exclude the missing submodule from the xclim.core, see:
+        exclude: ^docs/|^tests/|^xclim/sdba|^xclim/core/missing.py
   - repo: https://github.com/gitleaks/gitleaks
     rev: v8.21.2
     hooks:

CHANGELOG.rst

@@ -6,9 +6,14 @@ v0.54.0 (unreleased)
 --------------------
 Contributors to this version: Trevor James Smith (:user:`Zeitsperre`), Pascal Bourgault (:user:`aulemahal`), Éric Dupuis (:user:`coxipi`), Sascha Hofmann (:user:`saschahofmann`).
 
+New features and enhancements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* Python 3.9 coding conventions have been dropped in favour of Python 3.10+ conventions. (:pull:`1988`).
+
 Breaking changes
 ^^^^^^^^^^^^^^^^
 * The minimum required version of `dask` has been increased to `2024.8.1`. (:issue:`1992`, :pull:`1991`).
+* The docstrings of many `xclim` modules, classes, methods, and functions have been slightly adjusted to ensure stricter compliance with established `numpy` docstring conventions. (:pull:`1988`).
 
 Bug fixes
 ^^^^^^^^^
@@ -25,6 +30,7 @@ Internal changes
 * `streamflow` entry replaced with `q` in ``variables.yml``.  (:issue:`1912`, :pull:`1996`)
 * In order to address 403 (forbidden) request errors when retrieving data from GitHub via ReadTheDocs, the ``nimbus`` class has been modified to use an overloaded `fetch` method that appends a User-Agent header to the request. (:pull:`2001`).
 * Addressed a very rare race condition that can happen if `pytest` is tearing down the test environment when running across multiple workers. (:pull:`1863`).
+* The `numpydoc` linting tool has been added to the development dependencies, linting checks, and the `pre-commit` configuration. (:pull:`1988`).
 
 CI changes
 ^^^^^^^^^^

Makefile

@@ -53,16 +53,17 @@ clean-test: ## remove test and coverage artifacts
 	rm -fr .pytest_cache
 
 lint: ## check style with flake8 and black
-	black --check xclim tests
-	ruff check xclim tests
-	flake8 --config=.flake8 xclim tests
-	vulture xclim tests
-	nbqa black --check docs
-	blackdoc --check --exclude=xclim/indices/__init__.py xclim
-	blackdoc --check docs
-	codespell xclim tests docs
-	deptry .
-	yamllint --config-file=.yamllint.yaml xclim
+	python -m black --check xclim tests
+	python -m ruff check xclim tests
+	python -m flake8 --config=.flake8 xclim tests
+	python -m vulture xclim tests
+	python -m nbqa black --check docs
+	python -m blackdoc --check --exclude=xclim/indices/__init__.py xclim
+	python -m blackdoc --check docs
+	codespell .
+	python -m numpydoc lint xclim/*.py xclim/ensembles/*.py xclim/indices/*.py xclim/indicators/*.py xclim/testing/*.py
+	python -m deptry .
+	python -m yamllint --config-file=.yamllint.yaml xclim
 
 test: ## run tests quickly with the default Python
 	pytest
@@ -73,9 +74,9 @@ test-all: ## run tests on every Python version with tox
 	tox
 
 coverage: ## check code coverage quickly with the default Python
-	coverage run --source xclim -m pytest
-	coverage report -m
-	coverage html
+	python -m coverage run --source xclim -m pytest
+	python -m coverage report -m
+	python -m coverage html
 	$(BROWSER) htmlcov/index.html
 
 autodoc-obsolete: clean-docs ## create sphinx-apidoc files (obsolete)
@@ -105,10 +106,10 @@ servedocs: autodoc-custom-index ## generate Sphinx HTML documentation, including
 	$(MAKE) -C docs livehtml
 
 release: dist ## package and upload a release
-	flit publish dist/*
+	python -m flit publish dist/*
 
 dist: clean ## builds source and wheel package
-	flit build
+	python -m flit build
 	ls -l dist
 
 install: clean ## install the package to the active Python's site-packages

docs/notebooks/xclim_training/XCLIM_calculate_index-Exemple.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "This notebook will get you started on the use of `xclim` to subset netCDF arrays and compute climate indicators, taking advantage of parallel processing capabilities offered by `xarray` and `dask`. \n",
     "\n",
-    "`xarray` is a python package making it easy to work with n-dimensional arrays. It labels axes with their names (time, lat, lon, level) instead of indices (0,1,2,3), reducing the likelihood of bugs and making the code easier to understand. One of the key strengths of `xarray` is that it knows how to deal with non-standard calendars (I'm looking at you 360_days) and can easily resample daily time series to weekly, monthly, seasonal or annual periods.  Finally, `xarray` is tightly inegrated with `dask`, a package that can automatically parallelize operations.\n"
+    "`xarray` is a python package making it easy to work with n-dimensional arrays. It labels axes with their names (time, lat, lon, level) instead of indices (0,1,2,3), reducing the likelihood of bugs and making the code easier to understand. One of the key strengths of `xarray` is that it knows how to deal with non-standard calendars (I'm looking at you 360_days) and can easily resample daily time series to weekly, monthly, seasonal or annual periods.  Finally, `xarray` is tightly integrated with `dask`, a package that can automatically parallelize operations.\n"
    ]
   },
   {
@@ -317,7 +317,7 @@
    "source": [
     "## 7. xclim computations are *lazy*\n",
     "\n",
-    "Up until now we have ony created a schedule of tasks with a small preview, not done any actual computations. As mentionned above, writing the output to disk will trigger the cascade of computations on all the chunks. "
+    "Up until now we have only created a schedule of tasks with a small preview, not done any actual computations. As mentioned above, writing the output to disk will trigger the cascade of computations on all the chunks. "
    ]
   },
   {
@@ -443,7 +443,7 @@
    "source": [
     "#### Threshold indices\n",
     "\n",
-    "`xclim` unit handling also applies to threshold indicators. Users can provide threshold in units of choice and `xclim` will adjust automatically.  For example determining the number of days with tasmax > 20°C users can define a threshold input of '20 C' or '20 degC' even if input data is in Kelvin.  Alernatively users could send provide a threshold in Kelvin '293.15 K' (if they really wanted to)"
+    "`xclim` unit handling also applies to threshold indicators. Users can provide threshold in units of choice and `xclim` will adjust automatically. For example determining the number of days with tasmax > 20°C users can define a threshold input of '20 C' or '20 degC' even if input data is in Kelvin. Alternatively, users could send provide a threshold in Kelvin '293.15 K' (if they really wanted to)"
    ]
   },
   {
@@ -486,13 +486,6 @@
     "out3 = atmos.tx_days_above(dsTasmax_C.tasmax, thresh=\"293.15 K\", freq=\"MS\")\n",
     "print(\"\\n3. results using inputs in °C : threshold in K: \\n\\n\", out3.values)"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

docs/references.bib

@@ -442,7 +442,7 @@ @article{blazejczyk_introduction_2013
 	volume = {86},
 	url = {https://repository.lboro.ac.uk/articles/journal_contribution/An_introduction_to_the_Universal_Thermal_Climate_Index_UTCI_/9347024/1},
 	doi = {10.7163/GPol.2013.1},
-	abstract = {The assessment of the thermal environment is one ofthe main issues in bioclimatic research, and more than 100 simple bioclimatic indices have thus far been developed to facilitate it. However, most of these indices have proved to be of limited applicability, and do not portroy the actual impacts of thermal conditions on human beings. Indices derived from human heatbalance models (one- or two-node) have been found to offer a better representation of the environmental impact in question than do simple ones. Indeed, the new generation of multi-node models for human heat balance do allow full account to be taken of heat transfer and exchange, both within the human body and between the body surface and the surrounding air layer. In this paper, it is essential background information regarding the newly-developed Universal Thermal Climate Index UTCI that is presented, this in fact deriving from the Fiala multi-node model of human heatbalance. The UTCI is defined as the air temperature (Ta) of the reference condition causing the same model response as actual conditions. UTCI was developed in 2009 by virtue of international co-operation between leading experts in the areas of human thermophysiology, physiological modelling, meteorology and climatology. The necessary research for this had been conducted within the framework of a special commission of the International Society of Biometeorology (ISB) and European COST Action 730.},
+	abstract = {The assessment of the thermal environment is one of the main issues in bioclimatic research, and more than 100 simple bioclimatic indices have thus far been developed to facilitate it. However, most of these indices have proved to be of limited applicability, and do not portroy the actual impacts of thermal conditions on human beings. Indices derived from human heatbalance models (one- or two-node) have been found to offer a better representation of the environmental impact in question than do simple ones. Indeed, the new generation of multi-node models for human heat balance do allow full account to be taken of heat transfer and exchange, both within the human body and between the body surface and the surrounding air layer. In this paper, it is essential background information regarding the newly-developed Universal Thermal Climate Index UTCI that is presented, this in fact deriving from the Fiala multi-node model of human heatbalance. The UTCI is defined as the air temperature (Ta) of the reference condition causing the same model response as actual conditions. UTCI was developed in 2009 by virtue of international co-operation between leading experts in the areas of human thermophysiology, physiological modelling, meteorology and climatology. The necessary research for this had been conducted within the framework of a special commission of the International Society of Biometeorology (ISB) and European COST Action 730.},
 	language = {en},
 	number = {1},
 	urldate = {2022-07-29},

environment.yml

@@ -31,7 +31,7 @@ dependencies:
   # Testing and development dependencies
   - black ==24.10.0
   - blackdoc ==0.3.9
-  - bump-my-version >=0.27.0
+  - bump-my-version >=0.28.1
   - cairosvg
   - codespell ==2.3.0
   - coverage >=7.5.0
@@ -55,6 +55,7 @@ dependencies:
   - nc-time-axis >=1.4.1
   - netcdf4  # Required for some Jupyter notebooks
   - notebook
+  - numpydoc >=1.8.0
   - pandas-stubs >=2.2
   - pooch >=1.8.0
   - pre-commit >=3.7
@@ -64,7 +65,7 @@ dependencies:
   - pytest-cov >=5.0.0
   - pytest-socket >=0.6.0
   - pytest-xdist >=3.2
-  - ruff >=0.5.6
+  - ruff >=0.7.0
   - sphinx >=7.0.0
   - sphinx-autobuild >=2024.4.16
   - sphinx-autodoc-typehints

pyproject.toml

@@ -75,6 +75,7 @@ dev = [
   "nbconvert <7.14", # Pinned due to directive errors in sphinx. See: https://github.com/jupyter/nbconvert/issues/2092
   "nbqa >=1.8.2",
   "nbval >=0.11.0",
+  "numpydoc >=1.8.0",
   "pandas-stubs >=2.2",
   "pip >=24.2.0",
   "pooch >=1.8.0",
@@ -84,7 +85,7 @@ dev = [
   "pytest-cov >=5.0.0",
   "pytest-socket >=0.6.0",
   "pytest-xdist[psutil] >=3.2",
-  "ruff >=0.5.6",
+  "ruff >=0.7.0",
   "tokenize-rt >=5.2.0",
   "tox >=4.21.2",
   "tox-gh >=1.4.4",
@@ -161,7 +162,7 @@ values = [
 ]
 
 [tool.codespell]
-skip = 'xclim/data/*.json,docs/_build,docs/notebooks/xclim_training/*.ipynb,docs/references.bib,__pycache__,*.gz,*.nc,*.png,*.svg,*.whl'
+skip = '*xclim/data/*.json,*docs/_build,*docs/notebooks/xclim_training/*.ipynb,*docs/references.bib,*.gz,*.nc,*.png,*.svg,*.whl'
 ignore-words-list = "absolue,astroid,bloc,bui,callendar,degreee,environnement,hanel,inferrable,lond,nam,nd,ot,ressources,socio-economic,sie,vas"
 
 [tool.coverage.run]
@@ -247,6 +248,34 @@ module = [
 ]
 ignore_missing_imports = true
 
+[tool.numpydoc_validation]
+checks = [
+  "all", # report on all checks, except the below
+  "ES01", # "No extended summary found"
+  "EX01", # "No examples section found"
+  "GL06", # "Found unknown section \"{section}\""
+  "SA01", # "See Also section not found",
+  "SS01" # "No summary found"
+]
+# remember to use single quotes for regex in TOML
+exclude = [
+  # don't report on objects that match any of these regex
+  '\.undocumented_method$',
+  '\.__repr__$',
+  # any object starting with an underscore is a private object
+  '\._\w+'
+]
+override_SS05 = [
+  # override SS05 to allow docstrings starting with these words
+  '^Access ',
+  '^Assess ',
+  '^Days ',
+  '^Degree-days ',
+  '^Griffiths ',
+  '^Process ',
+  '^Statistics '
+]
+
 [tool.pytest.ini_options]
 minversion = "7.0"
 addopts = [

tests/test_indicators.py

@@ -591,7 +591,7 @@ def test_parse_doc():
     assert doc["notes"].startswith("Let")
     assert "math::" in doc["notes"]
     assert "references" not in doc
-    assert doc["long_name"] == "The mean daily temperature at the given time frequency"
+    assert doc["long_name"] == "The mean daily temperature at the given time frequency."
 
     doc = parse_doc(xclim.indices.saturation_vapor_pressure.__doc__)
     assert (

tests/test_sdba/test_adjustment.py

@@ -37,7 +37,12 @@
     get_correction,
     invert,
 )
-from xclim.testing.sdba_utils import nancov  # noqa
+
+
+def nancov(X):
+    """Numpy's cov but dropping observations with NaNs."""
+    X_na = np.isnan(X).any(axis=0)
+    return np.cov(X[:, ~X_na])
 
 
 class TestBaseAdjustment:

tox.ini

@@ -33,27 +33,21 @@ extras =
 deps =
     codespell ==2.3.0
     deptry==0.16.1
-    flake8 >=7.0.0
-    flake8-rst-docstrings
-    black[jupyter]==24.4.2
+    flake8==7.1.1
+    flake8-rst-docstrings==0.3.0
+    black[jupyter]==24.10.0
     blackdoc==0.3.9
-    nbqa
-    ruff==0.4.10
+    nbqa==1.8.2
+    numpydoc==1.8.0
+    ruff==0.7.0
     vulture==2.11
     yamllint==1.35.1
 commands_pre =
 commands =
-    black --check xclim tests
-    ruff check xclim tests
-    flake8 --config=.flake8 xclim tests
-    vulture xclim tests
-    nbqa black --check docs
-    blackdoc --check --exclude=xclim/indices/__init__.py xclim
-    blackdoc --check docs
-    codespell xclim tests docs
-    deptry .
-    yamllint --config-file=.yamllint.yaml xclim
+    make lint
 commands_post =
+allowlist_externals =
+    make
 
 [testenv:docs]
 description = Build the documentation with makefile under {basepython}