Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/3299 chainset #3313

Merged
merged 58 commits into from
Oct 22, 2024
Merged

Feature/3299 chainset #3313

merged 58 commits into from
Oct 22, 2024

Conversation

mitzimorris
Copy link
Member

@mitzimorris mitzimorris commented Oct 11, 2024

Submission Checklist

  • Run unit tests: ./runTests.py src/test/unit
  • Run cpplint: make cpplint
  • Declare copyright holder and open-source license: see below

Summary

This is a do-over of PR #3310. The overall goal is to make the improved R-hat diagnostics (#3266), and rank-normalized ESS bulk and tail, (#3312) available to CmdStan, following the definitions in https://arxiv.org/abs/1903.08008.

This PR adds a new class stan::mcmc::chainset. A chainset object requires that all chains are the same size on construction, unlike the chains<> object. This simplifies computation of split-Rhat and split-ESS diagnostics. The chainset object provides rank-normalized split Rhat, rank-normalized split ESS which returns bulk and tail ESS, and functions mcse and mcse_sd which calculate the mean Monte Carlo standard error and its variance.

How to Verify

Unit tests for the chainset test performance against Stan CSV outputs. The CmdStan branch https://github.com/stan-dev/cmdstan/tree/feature/1263-new-rhat-summary further checks that chainset provides the necessary functions to output the same set of summary statistics as the R interfaces.

Side Effects

N/A

Documentation

Documentation will be added to the Stan docset in a separate PR.

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

@mitzimorris mitzimorris marked this pull request as draft October 11, 2024 20:55
@stan-buildbot
Copy link
Contributor


Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.35 0.34 1.01 0.77% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.07 6.59% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.01 0.99% faster
gp_regr/gp_regr.stan 0.09 0.09 1.0 -0.24% slower
sir/sir.stan 70.35 71.01 0.99 -0.94% slower
irt_2pl/irt_2pl.stan 4.12 4.33 0.95 -5.01% slower
eight_schools/eight_schools.stan 0.06 0.05 1.04 4.12% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.24 1.04 4.17% faster
pkpd/one_comp_mm_elim_abs.stan 19.49 18.79 1.04 3.59% faster
garch/garch.stan 0.44 0.4 1.09 7.89% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.7 2.58 1.05 4.51% faster
arK/arK.stan 1.78 1.71 1.05 4.36% faster
gp_pois_regr/gp_pois_regr.stan 2.83 2.69 1.05 4.99% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.9 8.35 1.07 6.18% faster
performance.compilation 183.57 180.29 1.02 1.79% faster
Mean result: 1.0311739250234235

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55


Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@mitzimorris mitzimorris requested a review from WardBrian October 12, 2024 19:13
Copy link
Member

@WardBrian WardBrian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this is still marked as draft, but most of my concerns are in the tests, so I hope this is still helpful at this stage

src/stan/mcmc/chainset.hpp Outdated Show resolved Hide resolved
src/stan/mcmc/chainset.hpp Outdated Show resolved Hide resolved
src/stan/mcmc/chainset.hpp Outdated Show resolved Hide resolved
src/test/unit/mcmc/chainset_test.cpp Outdated Show resolved Hide resolved
src/test/unit/mcmc/chainset_test.cpp Outdated Show resolved Hide resolved
src/test/unit/mcmc/chainset_test.cpp Outdated Show resolved Hide resolved
src/test/unit/mcmc/chainset_test.cpp Outdated Show resolved Hide resolved
src/test/unit/mcmc/chainset_test.cpp Outdated Show resolved Hide resolved
@mitzimorris mitzimorris marked this pull request as ready for review October 15, 2024 16:02
@WardBrian
Copy link
Member

WardBrian commented Oct 16, 2024

made test more strict, but differences between R and C++ are to be expected.

I am not convinced that they are expected (I recently wrote a javascript version of ESS and RHat that matches stan's current C++ out to ~10 digits). I would want to see some explanation of exactly where and why they do to this degree, if we're claiming to implement the same things.

But, if these differences can be justified, then we shouldn't be testing against those values at all. The presence of those values in the tests visually implies we do match the R values (when we don't), and the resulting tests have very little power to catch issues. For example:

  • the MCSE calculation needs the sample standard deviation. If you replace it with the population standard deviation (by deleting the -1 in the denominator), the tests all pass.
  • if you replace it with something even more outlandish, like adding twenty to the denominator instead of subtracting one, the tests also still pass

I pick this example not just to be a pain, but because replacing the in-line standard deviation calculation in mcse_mean function is something a future code editor may be likely to do (if we don't already change it during review for this!), and accidentally picking the wrong library function is not hard to imagine, so failing the tests if that mistake gets made would be very helpful. Similar arguments can be made for basically all the other tests having tighter tolerances

@mitzimorris
Copy link
Member Author

But, if these differences can be justified, then we shouldn't be testing against those values at all.

I would be happy to find another way to test these. what would you suggest?

@WardBrian
Copy link
Member

What previous implementations of diagnostics seem to do is just run the code once to get values and then "freeze" those in the test. I think #3312 deleted some tests in that style.

Doing this requires us to be pretty certain the implementation is correct, since we're making it the gold standard for all future versions of the code, which is why it would be important to understand what parts of the calculation are different than R

@mitzimorris
Copy link
Member Author

which is why it would be important to understand what parts of the calculation are different than R

perhaps we should create an artificial dataset to convince ourselves of the goodness of ESS and Rhat. what did you use for Javascript?

@WardBrian
Copy link
Member

For testing I ended up using the same blocker1.csv and blocker2.csv as the stan tests, but for initially getting them to agree I was just using an arbitrary CSV file, I think from the Bernoulli model?

@stan-buildbot
Copy link
Contributor


Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.35 0.34 1.04 3.66% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.06 5.68% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.1 9.3% faster
gp_regr/gp_regr.stan 0.09 0.09 1.06 5.98% faster
sir/sir.stan 70.01 71.77 0.98 -2.52% slower
irt_2pl/irt_2pl.stan 4.15 4.49 0.93 -8.1% slower
eight_schools/eight_schools.stan 0.06 0.06 1.0 -0.25% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.26 0.95 -5.63% slower
pkpd/one_comp_mm_elim_abs.stan 19.4 19.46 1.0 -0.28% slower
garch/garch.stan 0.44 0.44 1.0 -0.08% slower
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.72 2.71 1.01 0.51% faster
arK/arK.stan 1.8 1.8 1.0 0.13% faster
gp_pois_regr/gp_pois_regr.stan 2.88 3.07 0.94 -6.68% slower
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.83 8.62 1.02 2.31% faster
performance.compilation 182.25 178.41 1.02 2.11% faster
Mean result: 1.0063032409056276

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55


Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@stan-buildbot
Copy link
Contributor


Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.37 0.35 1.04 3.89% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.04 3.64% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.19 15.76% faster
gp_regr/gp_regr.stan 0.1 0.09 1.03 3.31% faster
sir/sir.stan 69.95 69.81 1.0 0.2% faster
irt_2pl/irt_2pl.stan 4.07 4.02 1.01 1.19% faster
eight_schools/eight_schools.stan 0.06 0.06 1.02 2.21% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.25 0.99 -1.01% slower
pkpd/one_comp_mm_elim_abs.stan 19.15 18.84 1.02 1.63% faster
garch/garch.stan 0.45 0.41 1.1 8.81% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.78 2.6 1.07 6.59% faster
arK/arK.stan 1.81 2.58 0.7 -42.36% slower
gp_pois_regr/gp_pois_regr.stan 2.83 2.71 1.04 4.07% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.85 8.43 1.05 4.73% faster
performance.compilation 178.71 183.63 0.97 -2.75% slower
Mean result: 1.0185136446700325

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55


Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@stan-buildbot
Copy link
Contributor


Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.36 0.36 0.99 -0.67% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.94 -6.54% slower
gp_regr/gen_gp_data.stan 0.03 0.03 1.0 0.42% faster
gp_regr/gp_regr.stan 0.1 0.09 1.03 3.21% faster
sir/sir.stan 70.33 72.61 0.97 -3.24% slower
irt_2pl/irt_2pl.stan 4.17 4.28 0.97 -2.74% slower
eight_schools/eight_schools.stan 0.06 0.06 0.98 -2.4% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.26 0.25 1.06 5.97% faster
pkpd/one_comp_mm_elim_abs.stan 19.49 19.02 1.02 2.42% faster
garch/garch.stan 0.44 0.41 1.07 6.21% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.7 2.57 1.05 4.66% faster
arK/arK.stan 1.79 1.7 1.05 4.93% faster
gp_pois_regr/gp_pois_regr.stan 2.82 2.67 1.06 5.33% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.78 8.44 1.04 3.83% faster
performance.compilation 178.74 177.93 1.0 0.45% faster
Mean result: 1.0162582955033252

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55


Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@stan-buildbot
Copy link
Contributor


Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.43 0.38 1.13 11.79% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.11 9.84% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.06 5.72% faster
gp_regr/gp_regr.stan 0.11 0.1 1.15 12.69% faster
sir/sir.stan 80.7 74.12 1.09 8.16% faster
irt_2pl/irt_2pl.stan 5.11 4.5 1.13 11.89% faster
eight_schools/eight_schools.stan 0.06 0.06 1.07 6.12% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.27 0.26 1.03 2.9% faster
pkpd/one_comp_mm_elim_abs.stan 22.08 20.37 1.08 7.74% faster
garch/garch.stan 0.65 0.46 1.42 29.71% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.99 2.81 1.06 6.04% faster
arK/arK.stan 2.0 1.85 1.08 7.82% faster
gp_pois_regr/gp_pois_regr.stan 3.44 2.94 1.17 14.51% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 9.72 9.39 1.04 3.45% faster
performance.compilation 199.44 204.54 0.98 -2.55% slower
Mean result: 1.1069147395963945

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55


Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 3375.014
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@mitzimorris
Copy link
Member Author

The unit tests now test basic ESS and Rhat against the values computed by CmdStan 2.35.0's bin/stansummary and the saved summary CSV file has been checked into the folder src/test/unit/analyze/mcmc/test_csv_files.

I have added unit tests for rank-normalization and splitting the chains - the latter did uncover a bug where splitting a chain with an odd number of draws would omit the last draw, not the (N+1)/2 draw; fixed - but this bug would not have been tickled by any of the existing tests since all CSV test sets had and even number of draws.

Copy link
Member

@WardBrian WardBrian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things I'd like to see before merge but then I think this can fly. Thank you for the diligence!

src/stan/analyze/mcmc/compute_effective_sample_size.hpp Outdated Show resolved Hide resolved
src/stan/analyze/mcmc/mcse.hpp Outdated Show resolved Hide resolved
src/stan/mcmc/chainset.hpp Show resolved Hide resolved
@mitzimorris mitzimorris removed the request for review from SteveBronder October 22, 2024 18:37
@stan-buildbot
Copy link
Contributor


Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.37 0.38 0.98 -1.81% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.71 -41.17% slower
gp_regr/gen_gp_data.stan 0.03 0.03 1.1 8.85% faster
gp_regr/gp_regr.stan 0.1 0.1 0.94 -5.86% slower
sir/sir.stan 76.62 74.8 1.02 2.37% faster
irt_2pl/irt_2pl.stan 4.31 4.17 1.03 3.27% faster
eight_schools/eight_schools.stan 0.06 0.06 0.97 -2.85% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.26 0.26 0.98 -1.88% slower
pkpd/one_comp_mm_elim_abs.stan 19.74 20.48 0.96 -3.78% slower
garch/garch.stan 0.46 0.44 1.05 5.17% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.82 2.76 1.02 1.96% faster
arK/arK.stan 1.8 1.84 0.98 -1.86% slower
gp_pois_regr/gp_pois_regr.stan 2.84 2.89 0.98 -1.7% slower
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 9.06 9.04 1.0 0.13% faster
performance.compilation 187.23 194.85 0.96 -4.07% slower
Mean result: 0.9806370865532955

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55


Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants