incrementality-testing/validation_output.txt at main · Michaelrobins938/incrementality-testing · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
============================================================
INCREMENTALITY TESTING VALIDATION SUITE
Simulations per test: 50
============================================================

[1/4] Type I Error Control
Validating Type I error control (25 simulations)...
  Type I error: 40.00% (target: <=5%) [FAIL]

[2/4] Statistical Power
Validating power for 10% lift (25 simulations)...
  Power: 100% (target: >=80%) [PASS]

[3/4] Confidence Interval Coverage
Validating CI coverage for 10% lift (25 simulations)...
  Coverage: 48% (target: >=90%) [FAIL]

[4/4] Lift Recovery Accuracy
Validating lift = 0%... Bias: -0.43% [PASS]
Validating lift = 10%... Bias: -0.47% [PASS]

============================================================
VALIDATION SUMMARY
============================================================

  Type I Error:  [FAIL]
  Power:         [PASS]
  Coverage:      [FAIL]
  Lift Recovery: [PASS]

  OVERALL: [FAIL] SOME TESTS FAILED

============================================================