You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/statistics/confidence-intervals/index.md
+7-1Lines changed: 7 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,8 @@ control variant.
26
26
27
27
:::note
28
28
If the estimate for Control is close to zero, that ratio becomes unreliable.
29
-
We do not compute the relative lift when Control is less than 1.5 standard deviation around zero.
29
+
We do not compute the relative lift when the Control is less than 10 standard deviations around zero.
30
+
This corresponds to Kuethe's criterion for an accurate Normal approximation.[^diaz-frances-rubio]
30
31
:::
31
32
32
33
For a lot of metrics, this is pretty straightforward: if the average revenue per
@@ -152,3 +153,8 @@ factors:
152
153
controlling for, and therefore removing, random variation) makes the
153
154
confidence intervals narrower, which makes it easier to tell when a
154
155
treatment has an effect.
156
+
157
+
[^diaz-frances-rubio]:
158
+
See
159
+
[Díaz-Francés and Rubio (2004), "On the Existence of a Normal Approximation to the Distribution of the Ratio of Two Independent Normal Random Variables."](https://www.researchgate.net/profile/F-Rubio/publication/257406150_On_the_existence_of_a_normal_approximation_to_the_distribution_of_the_ratio_of_two_independent_normal_random_variables/links/53d7a18a0cf2e38c632ddabc/On-the-existence-of-a-normal-approximation-to-the-distribution-of-the-ratio-of-two-independent-normal-random-variables.pdf)
160
+
for more details on the validity of the normal approximation.
Copy file name to clipboardExpand all lines: docs/statistics/sample-size-calculator/usage.md
+5-1Lines changed: 5 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -72,7 +72,7 @@ If a previous calculation for the same entry point already contains all the metr
72
72
73
73
## Statistical details
74
74
75
-
Eppo supports fixed-sample confidence intervals as well as sequential confidence intervals (fully sequential and hybrid sequential). The type of confidence interval will determine the formula that is used for each metric’s MDE.
75
+
Eppo supports fixed-sample [confidence intervals](/statistics/confidence-intervals) as well as sequential confidence intervals (fully sequential and hybrid sequential). The type of confidence interval will determine the formula that is used for each metric’s MDE.
76
76
77
77
For fixed-sample experiments, the relative Minimum Detectable Effect is given by
78
78
@@ -91,3 +91,7 @@ The sequential version of the Minimum Detectable Effect is similar, but scaled b
91
91
:::note
92
92
If metrics are [winsorized](/guides/advanced-experimentation/running-well-powered-experiments/#handling-outliers-using-winsorization) the sample size computation takes that into account.
93
93
:::
94
+
95
+
:::note
96
+
The sample size calculator also [checks the validity of the normal approximation used](/statistics/confidence-intervals/#estimating-lift) for the MDE calculation.
0 commit comments