Skip to content

Commit 89b9e55

Browse files
Remove duplicate intros from theory chapter (#186)
* refactor: remove duplicate intros from theory chapter * enhance: add back main intro for mm to theory chapter
1 parent 50da67c commit 89b9e55

File tree

3 files changed

+7
-13
lines changed

3 files changed

+7
-13
lines changed

Chapters/Model_comparison.qmd

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -392,7 +392,11 @@ While the main function of LOO is to compare models, the values ​​of $k$ can
392392

393393
## When Pareto k values are too high {#sec-IWMM-theory}
394394

395-
The efficiency of PSIS-LOO-CV depends critically on stable importance sampling, which breaks down when the Pareto-$k$ diagnostic exceeds a threshold (typically around 0.7 for standard MCMC sample sizes), leading to unreliable estimates. The method **importance weighted moment matching (IWMM)** presented by @paananen_2020 offers a solution by transforming the posterior draws to improve the proposal distribution.
395+
The efficiency of PSIS-LOO-CV depends critically on stable importance sampling, which breaks down when the Pareto-$k$ diagnostic exceeds a threshold (typically around 0.7 for standard MCMC sample sizes), leading to unreliable estimates.
396+
397+
When observations produce high Pareto-$k$ values, we can use standard LOO-CV for those observations, i.e. refit the model while removing those problematic observations one at a time. This could be useful if the number of observations with high $\hat k$ is small, otherwise it can easily become too expensive for routine use. K-fold CV is another option but it can also be too expensive.
398+
399+
But those are not the only two solutions when PSIS-LOO-CV produces high $\hat k$ values. Instead of expensive model refitting, the method **importance weighted moment matching (IWMM)**, presented by @paananen_2020, works by improving the proposal distribution by iteratively transforming existing posterior draws to better match the moments of the target distribution.
396400

397401
To understand how moment matching improves these importance weights, we first need to understand which Monte Carlo estimator we are using.
398402

Chapters/Model_comparison_large_data.qmd

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,7 @@ decompress("../models/prerun/model_comparison_large_data_00.nc")
2424
decompress("../models/prerun/model_comparison_large_data_03.nc")
2525
```
2626

27-
Comparing Bayesian models on large datasets presents a computational challenge. Full leave-one-out cross-validation (LOO) requires computing $n$ posteriors and brute force K-fold cross-validation methods become prohibitively expensive as data size grows.
28-
29-
To overcome these challenges, we will use the approach described in @magnusson_2020 to combine fast approximations with targeted, exact computations. The key idea is to use cheap approximations, called *surrogates*, for all $n$ observations, and then correct these approximations using exact Pareto smoothed importance sampling (PSIS) LOO-CV on just a small random subsample with a *difference estimator* that corrects for the approximation error across the full dataset. For details of the method we recommend you to read @sec-subsampledloo-theory and references therein.
27+
In this chapter, we demonstrate how to efficiently compare Bayesian models on large datasets using subsampled PSIS-LOO-CV. We apply the methods introduced in @sec-subsampledloo-theory to a real dataset containing thousands of observations, showing how to balance computational efficiency with statistical accuracy. The workflow combines fast surrogate approximations with exact computations on a small subsample, and can be further accelerated using approximate posteriors from variational inference or Laplace approximations. For theoretical background and details of the method, we recommend you read @sec-subsampledloo-theory and references therein.
3028

3129
## Workflow for model comparison
3230

Chapters/Moment_Matching.qmd

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,7 @@ plt.rcParams["figure.dpi"] = 100
1515
SEED = 4711
1616
```
1717

18-
The efficiency of PSIS-LOO-CV depends critically on stable importance sampling, which breaks down when the Pareto-$k$ diagnostic exceeds a threshold (typically around 0.7 for standard MCMC sample sizes), leading to unreliable estimates.
19-
20-
When observations produce high Pareto-$k$ values, we can use standard LOO-CV for those observations, i.e. refit the model while removing those problematic observations one at a time. This could be useful if the number of observations with high $\hat k$ is small, otherwise it can easily become too expensive for routine use. K-fold CV is another option but it can also be too expensive.
21-
22-
But those are not the only two solution when PSIS-LOO-CV gives high $\hat k$ values. Instead of expensive model refitting. The method **importance weighted moment matching (IWMM)** presented by @paananen_2020 works by improving the proposal distribution by iteratively transforming existing posterior draws to better match the moments of the target distribution.
23-
24-
In this chapter we walk through a complete example showing how to use IWMM with ArviZ, for details on how this methods works we recommend @sec-IWMM-theory.
25-
26-
For this example, we'll fit a Poisson regression model to the `roaches` data from @gelman_hill_2007, which examines the efficacy of a pest management system at reducing cockroach infestations in urban apartments. The model is intentionally misspecified to demonstrate the benefits of moment matching when influential observations lead to high Pareto-$k$ values.
18+
In this chapter, we demonstrate how to apply importance weighted moment matching (IWMM) to improve PSIS-LOO-CV estimates when influential observations produce high Pareto-$k$ values. We walk through a complete applied example using the `roaches` dataset from @gelman_hill_2007, showing how to specify the required functions and apply moment matching to resolve problematic importance sampling approximations without expensive model refitting. The workflow demonstrates ArviZ's `loo_moment_match()` function with a deliberately misspecified Poisson regression model where multiple observations are flagged as problematic. For theoretical background and details of how moment matching works, we recommend you read @sec-IWMM-theory and references therein.
2719

2820
## Roaches data and Poisson regression model
2921

0 commit comments

Comments
 (0)