Effect of process noise and hierarchy measured by parameter retrieval quality (RQ)

[hyunjimoon]

RQ := retrieval quality of estimated parameter measured by SBC distance metric, as a function of precision and setting of a specific model

Research question

How does process noise affect RQ? Which dynamic model is hospitable for hierarchical model?

Research hypothesis:

draws2data

1. hierarchy and process noise have a similar effect on synthetic data generation (measured by observed time series)
2. hierarchizing (adding subscripts) process noise can be thought of as a two-layered hierarchy

data2draws

3. hierarchy and process noise have different effect on RQ in current stanify estimation, as process noise structure is ignored during estimation by updating process noise scale to zero during estimation

4. hierarchy and process noise may have similar effects on kalman/particle filtering estimation as process noise structure is not ignored during estimation

5. depending on the level of process noise scale designed for draws2data and data2draws:
   I) $\sigma_{draws2data}$ > $\sigma_{data2draws}$: fitting n+1 hierarchical model with n hierarchical model
   II) $\sigma_{draws2data}$ < $\sigma_{data2draws}$: fitting n hierarchical model with n+1 hierarchical model
   so process noise fills the gap between integer-interval hierarchy level. Using first-order autocorrelated process noise, the model may be gaussian process

6. the effect of process noise in observed data would be greater in reinforcement-loop dominant compared to the balance-loop dominant system

7. hierarchical model is more useful when the reinforcement loop is dominant

8. hierarchical model i.e. process noise may cause the transition from reinforcement to a balance-loop dominant system

@tomfid it may be a novel, but may I ask for quick feedback on the hypothesis?

Evidence or exploration plan

1, 3 : process noise would make time series murkier (1) so the closer the estimated parameter is located to where process noise is located, lower the RQ becomes: RQ_{prey_birth_frac} >RQ_{pred_birth_frac} from below (3).

3:

• hierarchical model make estimation worse: (RQ: pred_birth_frac > pred_birth_frac[region])
• effect of adding process noise and adding a hierarchical structure is similar (RQ_{pred_birth_frac} with larger $\sigma_p$ ~ RQ_{pred_birth_frac[region]})

adding hierarchy in stanify:

setting = {
    "est_param_names": ["prey_birth_frac", "pred_birth_frac"],
    "hier_est_param_names" : ["prey_birth_frac"], 
    "target_simulated_vector_names": ["prey", "predator"],
    "driving_vector_names": ["process_noise_uniform_driving"],
    "model_name": "prey_predator",
}
numeric = {
    "process_noise_uniform_driving": np.random.uniform(low=-.5, high=.5, size = precision['N']),
    'process_noise_scale': 0
}

adding process noise in stanify:

setting = {
    "est_param_names": ["prey_birth_frac"], 
    "hier_est_param_names" : [],
    "target_simulated_vector_names": ["prey", "predator"],
    "driving_vector_names": ["process_noise_uniform_driving"],
    "model_name": "prey_predator",
}
numeric = {
    "process_noise_uniform_driving": np.random.uniform(low=-.5, high=.5, size = precision['N']),
    'process_noise_scale': 1
}

6: process noise killed prey-predator oscillation

8: dynamic change of eigenvalue sign from Rogelio's structural dominance analysis paper