fairBS and fairRPS

[aaronspring]

Description

add fair brier score https://www-miklip.dkrz.de/about/problems/

• add as fair=False keyword into brier

• Same for rps

• name and cleanup each test in probabilistic (sorry for mixing two PRs into one, the purpose of tests was unclear to me so I gave all tests a docstring and refactored a bit)

• Question: With fair=True brier_score requires members in forecast, with brier_score(fair=False) it requires probabilities. Should we align this? Or at least allow also member in forecasts for brier_score(fair=False) and do forecast.mean('member') internally (my favorite but not implemented yet)?

Closes #162 #255

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance (if you touched existing code run asv to detect performance changes)
• refactoring

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. This could point to a cell in the updated notebooks. Or a snippet of code with accompanying figures here.

Checklist (while developing)

• I have added docstrings to all new functions.
• I have commented my code, particularly in hard-to-understand areas
• Tests added for pytest, if necessary.

Pre-Merge Checklist (final steps)

• I have rebased onto master or develop (wherever I am merging) and dealt with any conflicts.
• I have squashed commits to a reasonable amount, and force-pushed the squashed commits.

References

https://www-miklip.dkrz.de/about/problems/

[aaronspring]

source for the tests that the adjustment reduces skill value

[aaronspring]

Ideally this shouldn't be needed, I dont understand why without this line I get out-of-bounds results in tests

[aaronspring]

any thought on this? maybe some weird edge case in my tests @raybellwaves

[raybellwaves]

Couple of thoughts. Does it happen on the fair_bool=True tests? If so can move it inside a if fair:

curious is xr.apply_ufunc(np.clip, ... faster that two where statements
a.where(a < 1, 1).where(a > 0, 0) or is it better with dask objects

[raybellwaves]

I see test_rps_limits is a new test so it may have be an issue before this PR.

[aaronspring]

still I am not 100% confident with this

[raybellwaves]

Question: With fair=True brier_score requires members in forecast, with brier_score(fair=False) it requires probabilities. Should we align this? Or at least allow also member in forecasts for brier_score(fair=False) and do forecast.mean('member') internally (my favorite but not implemented yet)?

I think doing forecast.mean('member') makes sense. We do that for our example anyway brier_score = xs.brier_score(obs3>.5, (fct3>.5).mean('member'))

Nice work on the tests