Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change default RNG to boost::mixmax #3264

Merged
merged 2 commits into from
Feb 14, 2024
Merged

Conversation

WardBrian
Copy link
Member

Submission Checklist

  • Run unit tests: ./runTests.py src/test/unit
  • Run cpplint: make cpplint
  • Declare copyright holder and open-source license: see below

Summary

Closes #3256. This switches the pRNG used by default in the services and tests to be boost::mixmax, as recommended by the boost maintainers: boostorg/random#92

Intended Effect

Resolve both #3167 and stan-dev/cmdstan#1241, avoid other issues with RNG quality.

How to Verify

Side Effects

Seeds used in previous versions will have different meanings from this version on. We should note this in the release notes.

Additionally, by changing the type alias of rng_t introduced in #3263, this PR changes the ABI of the model base, so code which calls it directly will need to be aware. This is most obviously relevant for things which use the standalone-functions feature of stanc like cmdstanr and rstan, I believe. @andrjohns

Documentation

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):
Simons Foundation

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

@WardBrian
Copy link
Member Author

This caused a couple models to change results in the CmdStan Performance Tests:

  • stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse
    This model seems to intentionally be miss-specified? The settings we are comparing against yield less than 10 effective samples and Rhats of 1.7, 1.8, 1.4, and 1.2, and seem to be very highly seed dependent with the current RNG, so I'm not surprised?
  • stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data
    Unlike the above, this seems to sample fine (ESS ~1000, Rhat 1.0), but it also gets different answers depending on the seed with the current RNG.

I don't know enough about the history of the project and how these models and settings were picked, but so far I've yet to encounter them flagging something that wasn't a false alarm.

@SteveBronder
Copy link
Collaborator

Your results match up better actually than the current tests so idt this is an issue

FAIL: golds/stat_comp_benchmarks_benchmarks_low_dim_gauss_mix_collapse_low_dim_gauss_mix_collapse.gold param mu.1 not within (-0.301108939227 - -0.70400742885) / 0.620216676511786 < 0.3
FAIL: golds/stat_comp_benchmarks_benchmarks_low_dim_gauss_mix_collapse_low_dim_gauss_mix_collapse.gold param mu.2 not within (-0.24229668616 - 0.15116454648) / 0.635178602376801 < 0.3
FAIL: golds/stat_comp_benchmarks_benchmarks_low_dim_gauss_mix_collapse_low_dim_gauss_mix_collapse.gold param sigma.2 not within (1.033877479 - 1.096754202) / 0.154013580545209 < 0.3
'stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan' had fails '[('k.9', 68.023, 509.71930927251054, 350.409)]' and errors '[]'
'stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan' had fails '[('mu.1', -0.70400742885, 0.620216676511786, -0.301108939227), ('mu.2', 0.15116454648, 0.635178602376801, -0.24229668616), ('sigma.2', 1.096754202, 0.154013580545209, 1.033877479)]' and errors '[]'
mu[1] -2.8079e-01
mu[2] -2.6932e-01
sigma[1] 1.0254e+00
sigma[2] 1.0299e+00
theta 4.9645e-01

gen_gp_data is a weird one to fail but idt thats an issue

@WardBrian WardBrian force-pushed the refactor/actually-switch-rngs branch from 09a7d48 to ebd4a5c Compare February 8, 2024 21:48
@WardBrian WardBrian force-pushed the refactor/actually-switch-rngs branch from ebd4a5c to 3b75dc0 Compare February 13, 2024 19:50
Copy link
Collaborator

@SteveBronder SteveBronder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@WardBrian WardBrian merged commit c22f26a into develop Feb 14, 2024
2 of 3 checks passed
@WardBrian WardBrian deleted the refactor/actually-switch-rngs branch February 14, 2024 15:01
hsbadr added a commit to stan-dev/rstan that referenced this pull request Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move away from the boost::ecuyer1988 pRNG
2 participants