Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document problems with ppc_stat with stat="mean" #329

Merged
merged 3 commits into from
Aug 2, 2024

Conversation

jgabry
Copy link
Member

@jgabry jgabry commented Jul 31, 2024

closes #321

Adds section at the very top of the ppc_stat documentation (with NOTE in bold) that mentions the problem with stat="mean":

NOTE: Although the default test statistic is the mean, this is unlikely to detect anything interesting in most cases. In general we recommend using some other test statistic as discussed in Section 5 of Gabry et al. (2019).

@jgabry jgabry requested a review from avehtari July 31, 2024 18:33
@jgabry
Copy link
Member Author

jgabry commented Jul 31, 2024

@avehtari If you prefer I can add a warning or message in addition to making this prominent in the documentation, although maybe making it very clear in the doc is enough?

@avehtari
Copy link
Contributor

avehtari commented Aug 1, 2024

I think many users who have already been using ppc_stat probably don't re-read the doc, so for those a warning would be good, but if you think that is too much, at least we should include the comment in the NEWS and release announcement, too

@jgabry
Copy link
Member Author

jgabry commented Aug 1, 2024

I think many users who have already been using ppc_stat probably don't re-read the doc

Yeah that's probably true. I've updated the PR to add a message if stat="mean".

@jgabry
Copy link
Member Author

jgabry commented Aug 2, 2024

I think many users who have already been using ppc_stat probably don't re-read the doc

Yeah that's probably true. I've updated the PR to add a message if stat="mean".

When running ppc_stat, ppc_stat_2d, ppc_stat_freqpoly, or the _grouped versions of those functions using stat="mean" the user will see the following:

Screenshot 2024-08-02 at 12 26 45 PM

@avehtari Does that message look good to you? I think it's implied that they should therefore choose a different test statistic, so I don't think we need to say "we recommend choosing a different statistic". But do you want me to add something about where to find an explanation or this is enough? And do you think it's clear what "weak" means here?

@avehtari
Copy link
Contributor

avehtari commented Aug 2, 2024

I think this is fine. The documentation now refers to a text which explains things

@jgabry jgabry merged commit ce4f5d1 into master Aug 2, 2024
9 of 10 checks passed
@jgabry jgabry deleted the document-problem-with-stat-mean branch August 5, 2024 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add warning about using ppc_stat stat="mean"
2 participants