🐢Turtles all the way down: SBC and Hierarchical Bayesian

[hyunjimoon]

Simulation-based calibration has been criticized for being computation-heavy. My argument is some people may unfairly judge SBC's preventive maintenance role. Or fixing problems that never happened.

I don't see how I can well-deliver this need for phenomenology, simulation as the third pillar of science, though.

[tomfid]

I have only limited sympathy for arguments about computation heaviness.

If something is computationally heavy because the model is very complex and you only have time to run it a few times, that's very bad, because it will surely be poorly understood, with lurking robustness problems.

However, that's not the case here, because SBC is addressing this kind of problem by exercising the model a lot. So SBC is well-directed computation that increases quality.

[hyunjimoon]

"When and where to let go"
Problem with a long history I am exploring with Stan team for a long time e.g. excerpts from Andrew's Simulation-based calibration: Some challenges and directions for future research

First, in practice we can often learn a lot from just a single simulated dataset, that is, N=1. Modeling or computational problems are often so severe that they show up from just one simulation, for example we’ll get parameter estimates blowing up or getting stuck somewhere far away from where they should be. In my workflow, I’m often doing this sort of informal experimentation where I set the parameter vector to some reasonable value, simulate fake data, fit the model, and check that the parameter inferences are in the ballpark. It would be good to semi-automate this process so that it can be easy to do in the cmdstanR and cmdstanPy environments, as well as in rstanarm and brms

(a) Adjusting for the miscalibration that will arise when we’re no longer averaging over the prior (because the number of replication draws N is not large or because we want to focus on the posterior or some other area of interest of parameter space).

(btw, let it go and vs let it be and [vs into the unknown (frozen2 main song])