Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[obsolete] Update sampling output window #32

Closed
wants to merge 3 commits into from
Closed

Conversation

magland
Copy link
Collaborator

@magland magland commented May 31, 2024

This is a redo of #21

I took all those changes and put them on top of the new chain of changes - since the file sharing has been revamped.

See the discussion in the other PR. I think there are still some things to check/improve, but I think it would be great to merge this for now.

@magland magland requested review from jsoules and WardBrian May 31, 2024 18:54
@WardBrian
Copy link
Collaborator

I don't know much about the state of javascript testing, but I would feel a lot better if we had some tests on the ess/rhat functions at least. They're famously tricky.

For example, I just did a run where this is what is reported in the UI:

Parameter Mean MCSE StdDev 5% 50% 95% N_Eff N_Eff/s R_hat
lp__ 17.40 0.04792 1.472 14.61 17.79 18.97 944.2 4253 1.001
accept_stat__ 0.9308 0.001749 0.1099 0.7072 0.9733 1.000 3951 1.780e+4 1.021
stepsize__ 0.2542 0.01212 0.02797 0.2224 0.2716 0.2906 5.330 24.01 7.127e+12
treedepth__ 2.979 0.01316 0.7855 2.000 3.000 4.000 3561 1.604e+4 1.013
n_leapfrog__ 10.46 0.08561 5.148 3.000 11.00 15.00 3616 1.629e+4 1.013
divergent__ 0.000 NaN 0.000 0.000 0.000 0.000 NaN NaN NaN
energy__ -15.93 0.05853 1.897 -18.30 -16.26 -12.19 1050 4729 1.001
alpha -0.1038 0.001918 0.06552 -0.2122 -0.1033 -0.0009766 1167 5259 1.008
beta 1.252 0.0002855 0.01058 1.235 1.252 1.270 1374 6187 1.005
sigma 0.09158 0.0008137 0.02854 0.05753 0.08598 0.1465 1230 5541 1.003

Downloading the draws and then splitting into four csvs before feeding to stansummary produces

Parameter Mean MCSE StdDev 5% 50% 95% N_Eff N_Eff/s R_hat
lp__ 1.73637e+01 4.57725e-02 1.45807e+00 1.44671e+01 1.77336e+01 1.89540e+01 1.01473e+03 inf 1.00543e+00
accept_stat__ 9.20493e-01 2.70119e-02 1.37043e-01 6.45094e-01 9.73256e-01 1.00000e+00 2.57397e+01 inf 1.05173e+00
stepsize__ 2.66687e-01 4.07137e-02 5.76645e-02 1.91289e-01 2.61945e-01 3.53548e-01 2.00602e+00 inf 6.53902e+13
treedepth__ 3.02475e+00 9.96645e-02 8.10741e-01 2.00000e+00 3.00000e+00 4.00000e+00 6.61734e+01 inf 1.02400e+00
n_leapfrog__ 1.07520e+01 8.12053e-01 5.61590e+00 3.00000e+00 1.10000e+01 1.50000e+01 4.78268e+01 inf 1.03143e+00
divergent__ 0.00000e+00 nan 0.00000e+00 0.00000e+00 0.00000e+00 0.00000e+00 nan nan nan
energy__ -1.58774e+01 5.88228e-02 1.92866e+00 -1.82709e+01 -1.62244e+01 -1.21951e+01 1.07503e+03 inf 1.00534e+00
alpha -1.00380e-01 1.89720e-03 6.60711e-02 -2.07514e-01 -1.00932e-01 8.42572e-03 1.21281e+03 inf 1.00103e+00
beta 1.25135e+00 3.10310e-04 1.06537e-02 1.23415e+00 1.25148e+00 1.26825e+00 1.17871e+03 inf 1.00139e+00
sigma 9.19273e-02 8.43256e-04 2.92637e-02 5.75119e-02 8.51957e-02 1.47602e-01 1.20432e+03 inf 1.00410e+00

Ideally these would agree. I think a package which implemented these MCMC diagnostics and tested them against Stan's implementation would be a great service to the broader ecosystem, so it is probably worth pulling them out

@magland
Copy link
Collaborator Author

magland commented Jun 4, 2024

@WardBrian I agree. I propose we first make sure everything is good in bayes_kit in Python, and then match it exactly in .js, with tests. I did this with ess_imse for mcmc-monitor and ported the code over to stan-playground. But I'm sure it needs to be revisited (IDK if bayes_kit was finalized at that point)

@WardBrian
Copy link
Collaborator

Momentum on the bayes-kit project has pretty much stalled. The tests in Stan's C++ have also been used in e.g. the Rust mcmc package, but porting them to this environment may be a bit annoying

@magland magland changed the title Update sampling output window [obsolete] Update sampling output window Jun 7, 2024
@magland
Copy link
Collaborator Author

magland commented Jun 7, 2024

replaced by #46

@WardBrian WardBrian closed this Jun 7, 2024
@WardBrian WardBrian deleted the plots2 branch June 14, 2024 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants