Effect of alpha parameter on confidence intervals with bootstrapping #2545
-
Hi, I have a METviewer question regarding the role of the alpha parameter (under the "Common" tab, in the box labeled "Conf Interval Alpha") in the generation of confidence intervals (CIs) in the plot when using aggregation statistics and bootstrapping. After reading up on bootstrapping, my impression is that changing alpha should cause the CIs in a plot of a vx metric to change. My understanding is that alpha represents the percentile values from both the bottom and top ends of the bootstrapped (resampled) distribution to cut off. So setting alpha = 0.05 should put the CI endpoints at 5% and 95%, respectively, while setting it to 0.25 should cause the CI bar to become shorter and its endpoints to move to the 25% and 75% percentiles of the distribution. (I might be off by a factor of 2, i.e. maybe it's +/- alpha/2 that are used as the cutoffs; please let me know if so.) The particular metric I'm looking at is spread-skill of 2-m temperature. My xml plots the skill (RMSE), the spread, and the ratio spread/skill. I considered two cases: alpha = 0.05 and alpha = 0.25. I've attached the plots of the two cases as well as the xml I used to generate them. What I noticed is that the CIs on the skill and spread curves did not change with alpha, but the CIs on the ratio did. I am wondering if this is correct; it seems to be the CIs on all curves should change. Thanks for any clarification you can provide on this. Gerard |
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 3 replies
-
Hi @gsketefian. Our METviewer expert, @TatianaBurek, is out of the office currently. @TaraJensen is going to take a look and see if she is able to provide assistance. @TatianaBurek when you return if you're able to provide additional assistance, we would appreciate it. |
Beta Was this translation helpful? Give feedback.
-
There is a Github issue: Please direct any further questions and comments to the comments section of the issue. |
Beta Was this translation helpful? Give feedback.
-
@JohnHalleyGotway Thanks for looking into this and confirming that it's a bug. @bikegeek Thanks for opening the Github issue. I will use alpha = 0.05 for now until the bug is fixed. |
Beta Was this translation helpful? Give feedback.
-
Is this a 'blocker' issue for you, or can you generate your plots with the
default alpha for now?
Regards,
Minna
…On Wed, Apr 10, 2024 at 12:32 PM gsketefian ***@***.***> wrote:
@JohnHalleyGotway <https://github.com/JohnHalleyGotway> Thanks for
looking into this and confirming that it's a bug. @bikegeek
<https://github.com/bikegeek> Thanks for opening the Github issue. I will
use alpha = 0.05 for now until the bug is fixed.
—
Reply to this email directly, view it on GitHub
<#2545 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4UJHXN7NSEYT2DAMAIODTY4WAS3AVCNFSM6AAAAABF5Q5U2SVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TANZUHA4TE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
@bikegeek I think we're ok with the default value of 0.05 for now, but I'll tag @JeffBeck-NOAA, @michelleharrold, @mkavulich, and @willmayfield so they can chime in as well. Do you happen to have an estimate of how long it would take to get a bug fix in? |
Beta Was this translation helpful? Give feedback.
-
The fix in METcalcpy should be straightforward, it's the additional work in METviewer which will take more time. We don't have the budget for doing any further METplus development until June of this year. |
Beta Was this translation helpful? Give feedback.
-
METviewer does not require any additional work since the bug is in METcalcpy |
Beta Was this translation helpful? Give feedback.
-
John HG wasn't sure whether METviewer needed any changes. Nonetheless,
with the amount of work for the upcoming beta release, we don't have the
resources to fix this before our funding situation changes.
…On Thu, Apr 11, 2024 at 12:54 PM Tatiana Burek ***@***.***> wrote:
METviewer does not require any additional work since the bug is in
METcalcpy
—
Reply to this email directly, view it on GitHub
<#2545 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4UJHRDEKLQFR6ZKQYX6HTY43L5VAVCNFSM6AAAAABF5Q5U2SVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAOBXHEYTM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
There is a Github issue:
dtcenter/METcalcpy#371
Please direct any further questions and comments to the comments section of the issue.