look into mid-fit reporting of metrics

[arik-shurygin]

Often an MCMC fit stalled on one chain, or starts to diverge only on, it would be helpful to figure out a way we can report some posteriors from the chains as they come in, rather than all at once at the end.

Look into a way to flush MCMC output after every sample, also analyze the performance impacts of doing something like this (try to keep it light weight no more than 5% increase to runtime)

@SamuelBrand1 maybe you are aware of something like this in other frameworks? Is this common to do?

[SamuelBrand1]

Hey @arik-shurygin .

Yes this is common, e.g. in Turing.jl they have a callback module that outputs to to tensorboard https://turinglang.org/TuringCallbacks.jl/stable/

[arik-shurygin]

not sure if this kills this issue but I found this:

https://forum.pyro.ai/t/is-callback-supported-for-numpyro-nuts-sampler/6308

we currently don't run such low-level commands as he suggests as a workaround in our fitting script.
@kokbent how high priority is this functionality. Would it be worth redesigning our .infer() method to do this?

I have not dug in particularly deep to the implementation details and dont know how much of a headache it will be.