Skip to content

Commit 159b934

Browse files
authored
Add Kuethe's criterion to docs (#602)
1 parent 2cad3c4 commit 159b934

File tree

2 files changed

+12
-2
lines changed

2 files changed

+12
-2
lines changed

docs/statistics/confidence-intervals/index.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ control variant.
2626

2727
:::note
2828
If the estimate for Control is close to zero, that ratio becomes unreliable.
29-
We do not compute the relative lift when Control is less than 1.5 standard deviation around zero.
29+
We do not compute the relative lift when the Control is less than 10 standard deviations around zero.
30+
This corresponds to Kuethe's criterion for an accurate Normal approximation.[^diaz-frances-rubio]
3031
:::
3132

3233
For a lot of metrics, this is pretty straightforward: if the average revenue per
@@ -152,3 +153,8 @@ factors:
152153
controlling for, and therefore removing, random variation) makes the
153154
confidence intervals narrower, which makes it easier to tell when a
154155
treatment has an effect.
156+
157+
[^diaz-frances-rubio]:
158+
See
159+
[Díaz-Francés and Rubio (2004), "On the Existence of a Normal Approximation to the Distribution of the Ratio of Two Independent Normal Random Variables."](https://www.researchgate.net/profile/F-Rubio/publication/257406150_On_the_existence_of_a_normal_approximation_to_the_distribution_of_the_ratio_of_two_independent_normal_random_variables/links/53d7a18a0cf2e38c632ddabc/On-the-existence-of-a-normal-approximation-to-the-distribution-of-the-ratio-of-two-independent-normal-random-variables.pdf)
160+
for more details on the validity of the normal approximation.

docs/statistics/sample-size-calculator/usage.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ If a previous calculation for the same entry point already contains all the metr
7272

7373
## Statistical details
7474

75-
Eppo supports fixed-sample confidence intervals as well as sequential confidence intervals (fully sequential and hybrid sequential). The type of confidence interval will determine the formula that is used for each metric’s MDE.
75+
Eppo supports fixed-sample [confidence intervals](/statistics/confidence-intervals) as well as sequential confidence intervals (fully sequential and hybrid sequential). The type of confidence interval will determine the formula that is used for each metric’s MDE.
7676

7777
For fixed-sample experiments, the relative Minimum Detectable Effect is given by
7878

@@ -91,3 +91,7 @@ The sequential version of the Minimum Detectable Effect is similar, but scaled b
9191
:::note
9292
If metrics are [winsorized](/guides/advanced-experimentation/running-well-powered-experiments/#handling-outliers-using-winsorization) the sample size computation takes that into account.
9393
:::
94+
95+
:::note
96+
The sample size calculator also [checks the validity of the normal approximation used](/statistics/confidence-intervals/#estimating-lift) for the MDE calculation.
97+
:::

0 commit comments

Comments
 (0)