🌌 Multiverse analysis to prevent selective reporting + internal inconsistency

[hyunjimoon]

Solely on my experience of fitting the hierarchical Bayes model with Venism for four months, internal inconsistency and selection bias worries me. Orchestrating different versions of data (updated daily), models (up to V78), and optimization algorithms (vensim, March and September Vengine) is challenging for me in the Covid and job satisfaction project.

Contrary to covid model, job model is even more challenging as parameters are less tangible. e.g. discovering "too many people are dying" or prior knowledge "most cost is from death" prompted me to revise the error in the covid model, but I don't know if these cues can exist for the job model.

Another is selection bias, reporting only the successfully calibrated subgroup index or parameter. Randomizing the presented plot is one solution but this also is seed-controlled.

We can adopt the multiverse analysis detailed in Steegen16_TransparencyThruMultiverseAnal.pdf whose idea is active data construction i.e. construct data multiverse, improving consistency by endogenizing variability (e.g. Stanify's design choice of making data as data function). Table 1 illustrates choices of data construction. (reminds me of 2^4 type covering man-kind #97 (comment))

Table 1. Processing choices

1. Assessment of fertility (F)—high vs low.
(a) F1: high = cycle days 7–14; low = cycle days 17–25
(b) F2: high = cycle days 6–14; low = cycle days 17–27
(c) F3: high = cycle days 9–17; low = cycle days 18–25
(d) F4: high = cycle days 8–14; low = cycle days 1–7 and 15–28
(e) F5: high = cycle days 9–17; low = cycle days 1–8 and 18–28

2. Next menstrual onset (NMO)
(a) NMO1: reported start date previous menstrual onset + computed cycle length
(b) NMO2: reported start date previous menstrual onset + reported cycle length
(c) NMO3: reported estimate of next menstrual onset

3. Assessment of relationship status (R) (single vs relationship)
(a) R1: single = response options 1 and 2; relationship = response options 3 and 4
(b) R2: single = response option 1; relationship = response options 2, 3, and 4
(c) R3: single = response option 1; relationship = response options 3 and 4

4. Exclusion of women based on cycle length (ECL)
(a) ECL1: no exclusion based on cycle length
(b) ECL2: exclusion of participants with computed cycle length greater than 25 or less than 35 days
(c) ECL3: exclusion of participants with reported cycle length greater than 25 or less than 35 days

5. Exclusion of women based on certainty ratings of start dates of two previous menstrual periods (EC)
(a) EC1: no exclusion based on certainty ratings
(b) EC2: exclusion of participants who are not certain about at least one start date (i.e., sure less than 6)