Vensim MCMC

[hyunjimoon]

Time delay exists till HMC, so this summarize all comments for current Vensim MCMC

• psrf jumps right after burnin (because it resets)?

[hyunjimoon]

For hierarchical model, none of the team (city, world, service) perfectly recovered parameters (:= ground truth included in the posterior credible interval). However, the degree was more severe for population level parameter (hyperparameter), compared to unit level parameters

[tomfid]

Prospects for autodiff are not looking good, because it requires parallel computation of every equation's gradient, so you can chain rule all the way through the simulation. It's attractive to have, because it would also make the Kalman filter an order of magnitude faster, but it's a very big job.

An alternative might be stochastic HMC (SHMC) but I think the efficiency loss is significant. I think it might also be possible to add some gradient-based proposals to DREAM, which might be a lot like the idea of running a lot of short HMC chains.

[tomfid]

For hierarchical model, none of the team (city, world, service) perfectly recovered parameters (:= ground truth included in the posterior credible interval).

Was this an adversarial test, or did they know the truth?

[hyunjimoon]

Alban (urban model) shared his model which includes subscripts and .vpd. Will mail this to you. Thanks Alban!

[tomfid]

Thanks!

[hyunjimoon]

Does the following NN flow-based optimization method for ode system look promising to you? @tomfid
https://github.com/stefanradev93/BayesFlow/blob/master/docs/source/tutorial_notebooks/Linear%20ODE%20system.ipynb

[hyunjimoon]

Hazhir advised community might need persuasion that HMC adds value, so we discussed comparing "HMC one gradient computation time" vs "Vensim one simulation" might be useful, but I need to inspect i) how to get one gradient time ii) would it be uniform across all warmup iii) what calculation time would be the most valid comparison.

I wonder what benchmark is used for MCMC algorithms in general. https://mc-stan.org/docs/reference-manual/hmc.html would be the best initialization.

[hyunjimoon]

Differentiating the variability from before and after convergence with the variability from the target (posterior distribution) itself is important.

For next Vensim MCMC, default choices for the end of burn in period should be taken good care. Currently, it has some problem (e.g. After burnin period, vensim resets statistics (e.g. psrf))