Effect of alpha parameter on confidence intervals with bootstrapping

[gsketefian]

Hi,

I have a METviewer question regarding the role of the alpha parameter (under the "Common" tab, in the box labeled "Conf Interval Alpha") in the generation of confidence intervals (CIs) in the plot when using aggregation statistics and bootstrapping.

After reading up on bootstrapping, my impression is that changing alpha should cause the CIs in a plot of a vx metric to change. My understanding is that alpha represents the percentile values from both the bottom and top ends of the bootstrapped (resampled) distribution to cut off. So setting alpha = 0.05 should put the CI endpoints at 5% and 95%, respectively, while setting it to 0.25 should cause the CI bar to become shorter and its endpoints to move to the 25% and 75% percentiles of the distribution. (I might be off by a factor of 2, i.e. maybe it's +/- alpha/2 that are used as the cutoffs; please let me know if so.)

The particular metric I'm looking at is spread-skill of 2-m temperature. My xml plots the skill (RMSE), the spread, and the ratio spread/skill. I considered two cases: alpha = 0.05 and alpha = 0.25. I've attached the plots of the two cases as well as the xml I used to generate them. What I noticed is that the CIs on the skill and spread curves did not change with alpha, but the CIs on the ratio did. I am wondering if this is correct; it seems to be the CIs on all curves should change.

Thanks for any clarification you can provide on this.

Gerard
@JeffBeck-NOAA @michelleharrold @mkavulich @willmayfield @VanderleiVargas-NOAA

plot_ss_TMP2m_nostoch_stoch.xml.txt

[jprestop]

Hi @gsketefian. Our METviewer expert, @TatianaBurek, is out of the office currently. @TaraJensen is going to take a look and see if she is able to provide assistance. @TatianaBurek when you return if you're able to provide additional assistance, we would appreciate it.

[JohnHalleyGotway]

@gsketefian I took a look at this issue. I was able to replicate the behavior you describe. I re-created your plot with alpha values of 0.05 and 0.25. The confidence intervals for the latter should be significantly narrower than for the former. And that is the case for the two derived ratio curves, but not for the other RMSE and Spread lines.

FYI, it's easy to see the exact difference by downloading data from the R data tab. I created two plots, downloaded the R data for both, and then ran opendiff on my Mac laptop to see the differences line-by-line.

I did some digging into METcalcpy, whose job it is to compute statistics and confidence intervals. On line 102 of metcalcpy/bootstrap.py, I see in that the bootstrap_and_value(... alpha=0.05 ...) function computes 95% CI's by default. on line 751 of metcalcpy/agg_stat.py, I see that the default alpha CI value is being overridden by user settings (alpha=self.params['alpha']), as we'd expect. However the call to the same bootstrap_and_value() function on line 826, DOES NOT INCLUDE an alpha=... setting.

Having alpha specified in one call to bootstrap_and_value() and not in another may explain the variability in confidence intervals widths you've uncovered.

I'd definitely classify this as a bug and request that @bikegeek write up a bugfix issue for it and fix it in METcalcpy... and perhaps patch METviewer to include the fix as well.

@gsketefian, for the time being, I'd recommend sticking with the default alpha value of 0.05 (corresponding to a 95% confidence interval), until we can get this bug fixed.

[bikegeek]

There is a Github issue:
dtcenter/METcalcpy#371

Please direct any further questions and comments to the comments section of the issue.

[gsketefian]

@JohnHalleyGotway Thanks for looking into this and confirming that it's a bug. @bikegeek Thanks for opening the Github issue. I will use alpha = 0.05 for now until the bug is fixed.

[bikegeek]

Is this a 'blocker' issue for you, or can you generate your plots with the default alpha for now? Regards, Minna

…

On Wed, Apr 10, 2024 at 12:32 PM gsketefian ***@***.***> wrote: @JohnHalleyGotway <https://github.com/JohnHalleyGotway> Thanks for looking into this and confirming that it's a bug. @bikegeek <https://github.com/bikegeek> Thanks for opening the Github issue. I will use alpha = 0.05 for now until the bug is fixed. — Reply to this email directly, view it on GitHub <#2545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHXN7NSEYT2DAMAIODTY4WAS3AVCNFSM6AAAAABF5Q5U2SVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TANZUHA4TE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gsketefian]

@bikegeek I think we're ok with the default value of 0.05 for now, but I'll tag @JeffBeck-NOAA, @michelleharrold, @mkavulich, and @willmayfield so they can chime in as well.

Do you happen to have an estimate of how long it would take to get a bug fix in?

[michelleharrold]

Agreed that for now we can use 0.05 -- this is what we typically use, but I could definitely see us wanting to explore 0.01 or the like in the future!

[JeffBeck-NOAA]

In addition to the strictest values of statistical significance (0.01, 0.05), it would be nice to have some flexibility to assess alpha=0.1 or even 0.15 for some of our shorter retrospective simulations to see how that impacts things.

[bikegeek]

The fix in METcalcpy should be straightforward, it's the additional work in METviewer which will take more time. We don't have the budget for doing any further METplus development until June of this year.

[TatianaBurek]

METviewer does not require any additional work since the bug is in METcalcpy

[bikegeek]

John HG wasn't sure whether METviewer needed any changes. Nonetheless, with the amount of work for the upcoming beta release, we don't have the resources to fix this before our funding situation changes.

…

On Thu, Apr 11, 2024 at 12:54 PM Tatiana Burek ***@***.***> wrote: METviewer does not require any additional work since the bug is in METcalcpy — Reply to this email directly, view it on GitHub <#2545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHRDEKLQFR6ZKQYX6HTY43L5VAVCNFSM6AAAAABF5Q5U2SVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAOBXHEYTM> . You are receiving this because you were mentioned.Message ID: ***@***.***>