Merge branch 'aggregated_filters_for_trials' into dependabot/pip/pill…

…ow-10.3.0
NREL · Jul 5, 2024 · 2724c66 · 2724c66
2 parents 4a5e805 + ac3574f
commit 2724c66
Show file tree

Hide file tree

Showing 45 changed files with 13,383 additions and 9,537 deletions.
diff --git a/.codecov.yml b/.codecov.yml
@@ -0,0 +1,6 @@
+# Ignoring Paths
+# --------------
+# which folders/files to ignore
+ignore:
+  - setup.py
+  - versioneer.py
diff --git a/.flake8 b/.flake8
@@ -2,10 +2,17 @@
 # see https://flake8.pycqa.org/en/latest/user/options.html
 
 [flake8]
+# E203 is not PEP8 compliant https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#slices
+# Is excluded from flake8's own config https://flake8.pycqa.org/en/latest/user/configuration.html
+extend-ignore = E203
 max-line-length = 99
 max-doc-length = 99
 per-file-ignores =
     # rdtools.x.y imported but unused
     __init__.py:F401
     # invalid escape sequence '\s'
     versioneer.py:W605
+exclude =
+    docs
+    .eggs
+    build
diff --git a/.github/workflows/pytest.yaml b/.github/workflows/pytest.yaml
@@ -43,4 +43,8 @@ jobs:
         pip install --timeout=300 ${{ matrix.env }}
     - name: Test with pytest ${{ matrix.env }}
       run: |
-        pytest
+        python -m pytest --cov=./ --cov-report=xml
+    - name: Upload coverage reports to Codecov
+      uses: codecov/codecov-action@v4
+      env:
+        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
diff --git a/.gitignore b/.gitignore
@@ -35,3 +35,6 @@ rdtools.egg-info*
 .\#*
 
 *.pickle
+
+# ignore vscode settings
+.vscode/
diff --git a/README.md b/README.md
@@ -1,11 +1,14 @@
 <img src="./docs/sphinx/source/_images/logo_horizontal_highres.png" width="500" alt="RdTools logo"/>
 
-Master branch: 
-[![Build Status](https://github.com/NREL/rdtools/workflows/pytest/badge.svg?branch=master)](https://github.com/NREL/rdtools/actions?query=branch%3Amaster)  
+Master branch:
+[![Build Status](https://github.com/NREL/rdtools/workflows/pytest/badge.svg?branch=master)](https://github.com/NREL/rdtools/actions?query=branch%3Amaster)
 
-Development branch: 
+Development branch:
 [![Build Status](https://github.com/NREL/rdtools/workflows/pytest/badge.svg?branch=development)](https://github.com/NREL/rdtools/actions?query=branch%3Adevelopment)
 
+Code coverage:
+[![codecov](https://codecov.io/gh/NREL/rdtools/graph/badge.svg?token=K2HDjFkBws)](https://codecov.io/gh/NREL/rdtools)
+
 RdTools is an open-source library to support reproducible technical analysis of
 time series data from photovoltaic energy systems. The library aims to provide
 best practice analysis routines along with the building blocks for users to
@@ -33,8 +36,8 @@ and the specific DOI coresponding to that version from [Zenodo](https://doi.org/
 
 -  Michael G. Deceglie, Ambarish Nag, Adam Shinn, Gregory Kimball,
    Daniel Ruth, Dirk Jordan, Jiyang Yan, Kevin Anderson, Kirsten Perry,
-   Mark Mikofski, Matthew Muller, Will Vining, and Chris Deline
-   RdTools, version {insert version}, Compuer Software,
+   Mark Mikofski, Matthew Muller, Will Vining, and Chris Deline,
+   RdTools, version {insert version}, Computer Software,
    https://github.com/NREL/rdtools. DOI:{insert DOI}
 
 The underlying workflow of RdTools has been published in several places.
@@ -57,11 +60,11 @@ appropriate:
    Detection Techniques in AC Power Time Series," 2021 IEEE 48th Photovoltaic
    Specialists Conference (PVSC), pp. 1638-1643 2021, DOI: [10.1109/PVSC43889.2021.9518733](https://doi.org/10.1109/PVSC43889.2021.9518733).
 
-  
+
 ## References
 The clear sky temperature calculation, `clearsky_temperature.get_clearsky_tamb()`, uses data
-from images created by Jesse Allen, NASA’s Earth Observatory using data courtesy of the MODIS Land Group.  
-https://neo.sci.gsfc.nasa.gov/view.php?datasetId=MOD_LSTD_CLIM_M  
+from images created by Jesse Allen, NASA’s Earth Observatory using data courtesy of the MODIS Land Group.
+https://neo.sci.gsfc.nasa.gov/view.php?datasetId=MOD_LSTD_CLIM_M
 https://neo.sci.gsfc.nasa.gov/view.php?datasetId=MOD_LSTN_CLIM_M
 
 Other useful references which may also be consulted for degradation rate methodology include:

diff --git a/docs/TrendAnalysis_example_pvdaq4.ipynb b/docs/TrendAnalysis_example_pvdaq4.ipynb
diff --git a/docs/cods_example.ipynb b/docs/cods_example.ipynb
diff --git a/docs/degradation_and_soiling_example_pvdaq_4.ipynb b/docs/degradation_and_soiling_example_pvdaq_4.ipynb
diff --git a/docs/notebook_requirements.txt b/docs/notebook_requirements.txt
@@ -10,18 +10,18 @@ decorator==4.3.0
 defusedxml==0.7.1
 entrypoints==0.2.3
 html5lib==1.0.1
-ipykernel==4.8.2
-ipython==8.10.0
+ipykernel==6.29.4
+ipython==8.23.0
 ipython-genutils==0.2.0
 ipywidgets==7.3.0
 jedi==0.16.0
-Jinja2==3.0.0
+Jinja2==3.1.3
 jsonschema==2.6.0
 jupyter==1.0.0
-jupyter-client==6.1.7
-jupyter-console==6.4.0
-jupyter-core==4.11.2
-jupyterlab-pygments==0.2.2
+jupyter-client==8.6.1
+jupyter-console==6.6.3
+jupyter-core==5.7.2
+jupyterlab-pygments==0.3.0
 lxml==4.9.1
 MarkupSafe==2.0.0
 mistune==2.0.3
@@ -30,17 +30,17 @@ nbconvert==7.0.0
 nbformat==5.1.0
 nest-asyncio==1.5.5
 notebook==6.4.12
-numexpr==2.8.0
-pandocfilters==1.4.2
+numexpr==2.10.0
+pandocfilters==1.5.1
 parso==0.5.2
 pexpect==4.6.0
 pickleshare==0.7.5
 prometheus-client==0.3.0
-prompt-toolkit==3.0.30
+prompt-toolkit==3.0.43
 ptyprocess==0.6.0
 pycparser==2.20
 Pygments==2.15.0
-pyzmq==22.2.1
+pyzmq==26.0.2
 qtconsole==4.3.1
 Send2Trash==1.8.0
 simplegeneric==0.8.1
@@ -49,7 +49,7 @@ terminado==0.8.3
 testpath==0.3.1
 tinycss2==1.1.1
 tornado==6.3.3
-traitlets==5.0.0
+traitlets==5.14.3
 wcwidth==0.1.7
 webencodings==0.5.1
 widgetsnbextension==3.3.0
diff --git a/docs/sphinx/source/api.rst b/docs/sphinx/source/api.rst
@@ -9,14 +9,14 @@ Submodules
 ==========
 
 RdTools is organized into submodules focused on different parts of the data
-analysis workflow.  
+analysis workflow.
 
 .. autosummary::
    :toctree: generated/
 
    analysis_chains
    degradation
-   soiling   
+   soiling
    availability
    filtering
    normalization
@@ -41,8 +41,9 @@ Object-oriented end-to-end analysis
    analysis_chains.TrendAnalysis.plot_soiling_interval
    analysis_chains.TrendAnalysis.plot_soiling_monte_carlo
    analysis_chains.TrendAnalysis.plot_pv_vs_irradiance
-
-
+   analysis_chains.TrendAnalysis.plot_degradation_timeseries
+
+
 Degradation
 ===========
 
@@ -71,6 +72,10 @@ Soiling
    annual_soiling_ratios
    SRRAnalysis
    SRRAnalysis.run
+   soiling_cods
+   CODSAnalysis
+   CODSAnalysis.iterative_signal_decomposition
+   CODSAnalysis.run_bootstrap
 
 
 System Availability
@@ -81,7 +86,7 @@ System Availability
 
 .. autosummary::
    :toctree: generated/
-   
+
    AvailabilityAnalysis
    AvailabilityAnalysis.run
    AvailabilityAnalysis.plot
@@ -95,15 +100,22 @@ Filtering
 
 .. autosummary::
    :toctree: generated/
-    
+
    clip_filter
    quantile_clip_filter
    logic_clip_filter
    xgboost_clip_filter
+   clearsky_filter
    csi_filter
+   pvlib_clearsky_filter
    poa_filter
    tcell_filter
    normalized_filter
+   two_way_window_filter
+   insolation_filter
+   hampel_filter
+   directional_tukey_filter
+   hour_angle_filter
 
 
 Normalization
@@ -120,7 +132,6 @@ Normalization
    irradiance_rescale
    normalize_with_expected_power
    normalize_with_pvwatts
-   normalize_with_sapm
    pvwatts_dc_power
    sapm_dc_power
    delta_index
@@ -161,8 +172,10 @@ Plotting
    :toctree: generated/
 
    degradation_summary_plots
+   degradation_timeseries_plot
    soiling_monte_carlo_plot
    soiling_interval_plot
    soiling_rate_histogram
    availability_summary_plots
-   tune_filter_plot
+   tune_filter_plot
+
diff --git a/docs/sphinx/source/changelog.rst b/docs/sphinx/source/changelog.rst
@@ -1,10 +1,14 @@
 RdTools Change Log
 ==================
+.. include:: changelog/pending.rst
+.. include:: changelog/v2.2.0-beta.2.rst
+.. include:: changelog/v2.2.0-beta.1.rst
 .. include:: changelog/v2.1.8.rst
 .. include:: changelog/v2.1.7.rst
 .. include:: changelog/v2.1.6.rst
 .. include:: changelog/v2.1.5.rst
 .. include:: changelog/v2.1.4.rst
+.. include:: changelog/v2.2.0-beta.0.rst
 .. include:: changelog/v2.1.3.rst
 .. include:: changelog/v2.1.2.rst
 .. include:: changelog/v2.1.1.rst

diff --git a/docs/sphinx/source/changelog/pending.rst b/docs/sphinx/source/changelog/pending.rst
@@ -0,0 +1,37 @@
+*******
+pending
+*******
+
+Breaking changes
+------------
+These changes have the potential to change answers in existing scripts
+when compared with older versions of RdTools
+
+* Use the pvlib method for clear sky detection by default in :py:func:`~rdtools.analysis_chains.TrendAnalysis` (:pull:`412`)
+
+Enhancements
+------------
+* Added a new wrapper function for clearsky filters (:pull:`412`)
+* Improve test coverage, especially for the newly added filter capabilities (:pull:`413`)
+* Added codecov.yml configuration file (:pull:`420`)
+
+Bug fixes
+---------
+* Fix typos in citation section of the readme file (:issue:`414`, :pull:`421`)
+
+Requirements
+------------
+* Specified versions in ``requirements.txt`` and ``docs/notebook_requirements.txt`` have been updated (:pull:`412`)
+* Increase maximum version of pvlib to <0.12 (:pull:`423`)
+* Update fonttools version to 4.43.0 in ``requirements.txt`` (:pull:`404`)
+* Update jinja2 from 3.0.0 to 3.1.3 in ``notebook_requirements.txt`` (:pull:`405`)
+
+Deprecations
+------------
+* Removed  :py:func:`~rdtools.normalization.sapm_dc_power` (:pull:`419`)
+* Removed  :py:func:`~rdtools.normalization.normalize_with_sapm` (:pull:`419`)
+
+Contributors
+------------
+* Martin Springer (:ghuser:`martin-springer`)
+* Michael Deceglie (:ghuser:`mdeceglie`)
diff --git a/docs/sphinx/source/changelog/v2.1.0.rst b/docs/sphinx/source/changelog/v2.1.0.rst
@@ -1,6 +1,6 @@
-************************
+***************************
 v2.1.0 (September 17, 2021)
-************************
+***************************
 
 API Changes
 -----------

diff --git a/docs/sphinx/source/changelog/v2.2.0-beta.0.rst b/docs/sphinx/source/changelog/v2.2.0-beta.0.rst
@@ -0,0 +1,33 @@
+**********************************
+v2.2.0-beta.0 (September 14, 2022)
+**********************************
+
+Enhancements
+------------
+* Specifying ``detailed=True`` in :py:func:`rdtools.plotting.degradation_summary_plots`
+  now shows the number of year-on-year slopes in addition to color coding points
+  (:issue:`298`, :pull:`324`)
+* The Combined estimation Of Degradation and Soiling (CODS) algorithm is implemented
+  in the soiling module and illustrated in an example notebook (:pull:`150`, :pull:`333`)
+* Circular block bootstrapping added as a method for calculating uncertainty in
+  ``degradation_year_on_year()`` via the ``Uncertainty_method`` argument (:pull:`150`)
+* New plotting functions :py:func:`rdtools.analysis_chains.TrendAnalysis.plot_degradation_timeseries` 
+  in TrendAnalysis object and :py:func:`rdtools.plotting.degradation_timeseries_plot`
+  to plot time series degradation trend.  (:issue:`334`, :pull:`335`)
+
+Testing
+-------
+* Added a CI notebook check (:pull:`270`)
+
+Requirements
+------------
+* Upgrade the notebook environment from python 3.7 to python 3.10.
+  (:issue:`319`, :pull:`326`)
+* Bump ``sphinx`` version from 3.2 to 4.5 and ``nbsphinx`` version
+  from 0.8.5 to 0.8.8 in the optional ``[doc]`` requirements (:pull:`317`, :pull:`325`)
+* ``arch`` and ``filterpy`` added as dependencies (:pull:`150`)
+* minimum version of ``numpy`` increased to 1.16 and minimum version of
+  statsmodels increased to 0.11.1 (:pull:`150`)
+* A number of updates to the environments specified in ``requirements.txt``,
+  ``requirements-min.txt``, and ``docs/notebook_requirements.txt``
+  (:pull:`326`, :pull:`314`, :pull:`337`)
diff --git a/docs/sphinx/source/changelog/v2.2.0-beta.1.rst b/docs/sphinx/source/changelog/v2.2.0-beta.1.rst
@@ -0,0 +1,7 @@
+********************************
+v2.2.0-beta.1 (December 7, 2022)
+********************************
+
+Enhancements
+------------
+* Added framework for running aggregated filters in ``analysis_chains.py`` (:pull:`348`)
diff --git a/docs/sphinx/source/changelog/v2.2.0-beta.2.rst b/docs/sphinx/source/changelog/v2.2.0-beta.2.rst
@@ -0,0 +1,23 @@
+********************************
+v2.2.0-beta.2 (December 1, 2023)
+********************************
+
+Enhancements
+------------
+* Return CODS results without bootstrapping when soiling signal
+  is small but raise warning ``soiling.py`` (:issue:`367` :pull:`400`)
+
+Bug fixes
+---------
+* Fix flake8 missing whitespaces ``bootstrap_test.py``, ``soiling_cods_test.py`` (:pull:`400`)
+* Specify dtype for seasonal samples ``soiling.py`` (:pull:`400`)
+* Update deprecated `check_less_precise` to `rtol` ``soiling_cods_test.py`` (:pull:`400`)
+
+Requirements
+------------
+* Bump arch to 5.6.0 in ``requirements.txt``
+
+Contributors
+------------
+* Martin Springer (:ghuser:`martin-springer`)
+* Michael Deceglie (:ghuser:`mdeceglie`)
diff --git a/docs/sphinx/source/examples.rst b/docs/sphinx/source/examples.rst
@@ -25,3 +25,4 @@ This page shows example usage of the RdTools analysis functions.
     examples/degradation_and_soiling_example_pvdaq_4
     examples/TrendAnalysis_example_pvdaq4
     examples/system_availability_example
+    examples/cods_example
diff --git a/docs/sphinx/source/examples/cods_example.nblink b/docs/sphinx/source/examples/cods_example.nblink
@@ -0,0 +1,3 @@
+{
+    "path": "../../../cods_example.ipynb"
+}
diff --git a/docs/sphinx/source/index.rst b/docs/sphinx/source/index.rst
@@ -327,6 +327,5 @@ Indices and tables
 
 
 .. links and references
-
 .. _release: https://github.com/NREL/rdtools/releases
 .. _github: https://github.com/NREL/rdtools