EVV 0.5.2

We are pleased to present EVV 0.5.2

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes. Several changes are contained in this release.

Statistical changes

#15 Switch to the Benjamini and Hochberg (1995) correction over the Benjamini and Yekutieli (2001) correction

Interface changes

#15 Output the uncorrected p-values and the number of rejected null hypotheses based on them to the output webpage

Full Changelog: v0.5.1...v0.5.2

EVV 0.5.1

We are pleased to present EVV 0.5.1

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes. Several changes are contained in this release.

Bugfixes

• #13 Fixes issue where joining the columns to a DataFrame in the TSC test broke with Pandas >v2.0

System updates

• #14 Adds automated testing and releasing

Full Changelog: v0.5.0...v0.5.1

EVV 0.5.0

We are pleased to present EVV 0.5.0

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes. Several changes are contained in this release.

Major updates

• Implement Benjamini and Yekutieli (2001) method for false discovery rate correction in the MVK test
  • This takes advantage of multiple concurrent tests to reduce erroneous rejection of null hypotheses
  • P-values are adjusted then compared against the global significance level (0.05 by default), if any are below the threshold, the tests are considered climate changing (i.e. the ensembles are different)

Minor improvements

• Add argument for output figure file format
• Use combined histogram plot for KS-ocean and KS-atmosphere tests
• Add sort-able tables to output

Bug fixes

• Cast arrays to float, fix bug for loading new PGN data
• Fix doubly defined function

Full Changelog: v0.4.0...v0.5.0

EVV 0.4.0

We are pleased to present EVV 0.4.0

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes.

Major changes

• Add Kolmogorov-Smirnov multi-variate test for MPAS-Ocean
  • Performs K-S test at each grid cell for multiple variables
  • Uses false discovery rate correction to compensate for overconfidence
• Add multi-instance ensemble processing for MPAS output (tested with ocean and sea ice)

Additional changes:

• Adds statsmodels dependency
• Visual tweaks to output plots

Full Changelog: v0.3.2...v0.4.0

EVV 0.3.2

We are pleased to present EVV 0.3.1

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes.

In this release, we actually fix E3SM issue 4759

This makes the TSC test match the original intent of the time step convergence (TSC) test. First, time steps are assessed for pass/fail, then an overall pass/fail is given.

• For a time step to fail, at least one variable has its null hypothesis rejected (i.e. RMSD difference has a p-value less than the threshold).
• For overall FAIL, at least two time steps within the inspection window must fail.

Additional improvements:

• Enhance TSC plots with threshold lines and caption
• Revise word order in figure captions
• Add pool_size argument for debugging mode

Full Changelog: v0.3.1...v0.3.2

EVV 0.3.1

We are pleased to present EVV 0.3.1

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes.

• Fix TSC test to match original pass/fail method
  Per E3SM issue 4759 (E3SM-Project/E3SM#4759)
  The TSC test should fail only if all points between 300s - 600s are
  below the failure threshold (p_threshold). This also fixes the plotting
  of P_min so that the variable threshold is plotted rather than a fixed
  0.5%

• Update copyright date

• Bump version number

• Fix readme formatting

EVV 0.3.0

We are pleased to present EVV 0.3.0

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes.

Changelog:

• Remove Python 2 support
• Update MVK, PGN, and TSC modules for LIVVkit 3 interface
• Minor improvements to data presentation for MVK test:
  • Add variable description to figures
  • Groups variables into accept/reject groups for figures and table

EVV 0.2.5

We are pleased to present EVV 0.2.5

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes.
Changelog:

• Bugfix for TSC test looking for baseline_component configuration argument

EVV 0.2.4

We are pleased to present EVV 0.2.3

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes.

Changelog:

• Add flexibility for model component names

  • Make the rmse writer function more general
    The CIME SystemTest pgn.py will write the init_model variable so that
    the initial condition file template can be adjusted as needed. This is
    used in rmse_writer to send output to comp_cld.nc

  • Add model component to arguments
    This will add the command line / config argument for the model
    component so that the input file template can be adjusted from the
    json config or command line args, so this is adaptable to E3SM and CESM

EVV 0.2.3

We are pleased to present EVV 0.2.3

EVV provides a number of statistical climate reproducibility tests of Earth system models for non-bit-for-bit
changes.

Changelog:

• Update E3SM ensemble module and extensions to use "eam" for the atmosphere model name in place of "cam"