effective sample size larger than number of samples

[ismael-mendoza]

Perhaps this is not specific to blackjax but any tips or resources would be greatly appreciated. I find that when I run NUTS for my problem with blackjax to get samples and I estimate their effective sample size with blackjax.diagnostics.effective_sample_size sometimes the output is larger than the total number of samples. I am using 5 chains with different random seeds initialized different positions (small ball around the true parameters) targeting the same target density and each outputs 1000 samples.

I have read in the stan manual that this can happen because the autocorrelation can be negative at certain lags and I think the blackjax implementation of ESS follows the strategy listed there to mitigated this. But I'm still consistently seeing larger ESS than total number of samples for all the parameters (my target density has dimensionality 6).

I was wondering if this is a known issue and if there is any strategy I could use to mitigate this or to obtain a conservative estimate of relative ESS that is below 1 ? Thanks in advance for any suggestions!

What I'm considering right now is simply to truncate it at the total number of samples, but I'm not sure if I an safely assume that this output means that ESS >= total # samples?

[ismael-mendoza]

also I suspect this is the case but I just wanted to be certain as the docstring does not mention it explicitly. Does the ESS calculation in blackjax use this equation for rho-hat from the stan manual?

[junpenglao]

Yes the ESS estimation in Blackjax follows the implementation in Stan (or Arviz), which will returns ESS > # samples when you are using a highly efficient sampler that produce anti-correlate samples. This is a feature not a bug.

[AdrienCorenflos]

I'm sure @junpenglao knows this but for @ismael-mendoza -> https://statmodeling.stat.columbia.edu/2018/01/18/measuring-speed-stan-incorrectly-faster-thought-cases-due-antithetical-sampling/

Btw, I'm not convinced by this blog post but the people who wrote it are more informed than me soooo.

[ismael-mendoza]

Thanks @junpenglao and @AdrienCorenflos for the answer and resources. So I think my takeaway after reading through more stuff and your helpful comments is that this effect is real, and that indeed we do have more "effective" samples in so far as the posterior mean (i.e error on the estimated posterior mean using chain samples will be lower than if we had used independent samples from the posterior), but this does not necessarily hold for other posterior quantities. Is that a reasonable interpretation? thanks again!

[ColCarroll]

Note that Hoffman, Sountsov, Radul mention that the expected squared jump distance (which NUTS maximizes) may focus too much on estimating means at the expense of other moments (section 2 here)

These too-long trajectories
do extra work to generate estimates of the highest
variance parameters that are better than the estimates
of lower-variance parameters; if we want to minimize
the maximum autocorrelation across dimensions, this is
a waste of effort. ESJD is also fooled by long trajectories’
ability to produce anti-correlated samples, which
are good for estimating means but bad for estimating
variances.