Skip to content

Commit c8649c6

Browse files
committed
add confidence interval field descriptions
1 parent c5d5d83 commit c8649c6

File tree

1 file changed

+26
-22
lines changed

1 file changed

+26
-22
lines changed

specs/abtesting-v3/common/schemas/Variant.yml

Lines changed: 26 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -6,29 +6,29 @@ variants:
66
The first variant is your _control_ index, typically your production index.
77
All of the additional variants are indexes with changed settings that you want to test against the control.
88
items:
9-
$ref: "#/variant"
9+
$ref: '#/variant'
1010

1111
variant:
1212
type: object
1313
additionalProperties: false
1414
properties:
1515
description:
16-
$ref: "../parameters.yml#/description"
16+
$ref: '../parameters.yml#/description'
1717
estimatedSampleSize:
1818
type: integer
1919
description: |
2020
Estimated number of searches required to achieve the desired statistical significance.
2121
22-
The A/B test configuration must include a `mininmumDetectableEffect` setting for this number to be included in the response.
22+
The A/B test configuration must include a `minimumDetectableEffect` setting for this number to be included in the response.
2323
example: 0
2424
index:
25-
$ref: "../parameters.yml#/index"
25+
$ref: '../parameters.yml#/index'
2626
trafficPercentage:
27-
$ref: "../parameters.yml#/trafficPercentage"
27+
$ref: '../parameters.yml#/trafficPercentage'
2828
metrics:
29-
$ref: "#/metrics"
29+
$ref: '#/metrics'
3030
metadata:
31-
$ref: "#/variantMetadata"
31+
$ref: '#/variantMetadata'
3232

3333
required:
3434
- index
@@ -38,9 +38,9 @@ variant:
3838

3939
metrics:
4040
type: array
41-
description: All ABTest metrics that were defined during test creation
41+
description: All ABTest metrics that were defined during test creation.
4242
items:
43-
$ref: "#/metric"
43+
$ref: '#/metric'
4444

4545
metric:
4646
type: object
@@ -56,24 +56,28 @@ metric:
5656
valueCIHigh:
5757
type: number
5858
format: double
59+
description: |
60+
The upper bound of the 95% confidence interval for the metric value. The confidence interval is calculated using
61+
either the relative ratio or relative difference between the metric values for the control and the variant.
62+
Relative ratio is used for metrics that are ratios (e.g., click-through rate, conversion rate),
63+
while relative difference is used for continuous metrics (e.g., revenue).
5964
valueCILow:
6065
type: number
6166
format: double
67+
description: |
68+
The lower bound of the 95% confidence interval for the metric value. The confidence interval is calculated using
69+
either the relative ratio or relative difference between the metric values for the control and the variant.
70+
Relative ratio is used for metrics that are ratios (e.g., click-through rate, conversion rate),
71+
while relative difference is used for continuous metrics (e.g., revenue).
6272
pValue:
6373
type: number
6474
format: double
6575
description: PValue for the first variant (control) will always be 0. For the other variants, pValue is calculated for the current variant based on the control.
66-
pValueCIHigh:
67-
type: number
68-
format: double
69-
pValueCILow:
70-
type: number
71-
format: double
7276
dimension:
7377
type: string
74-
description: Dimension defined during test creation
78+
description: Dimension defined during test creation.
7579
metadata:
76-
$ref: "#/metricMetadata"
80+
$ref: '#/metricMetadata'
7781
required:
7882
- name
7983
- updatedAt
@@ -93,24 +97,24 @@ metric:
9397
updatedAt: 2025-05-15T17:52:15.644906Z
9498
value: 1200.50
9599
pValue: 0.04
96-
metadata: { "winsorizedValue": 80.2 }
100+
metadata: {'winsorizedValue': 80.2}
97101
- name: revenue
98102
dimension: EUR
99103
updatedAt: 2025-05-15T17:52:15.644906Z
100104
value: 999.66
101105
pValue: 0.04
102-
metadata: { "winsorizedValue": 888.8 }
106+
metadata: {'winsorizedValue': 888.8}
103107

104108
variantMetadata:
105109
type: object
106-
description: Variant specific metadata
110+
description: Variant specific metadata.
107111
properties:
108112
filterEffects:
109-
$ref: "../parameters.yml#/filterEffects"
113+
$ref: '../parameters.yml#/filterEffects'
110114

111115
metricMetadata:
112116
type: object
113-
description: Metric specific metadata
117+
description: Metric specific metadata.
114118
properties:
115119
winsorizedValue:
116120
type: number

0 commit comments

Comments
 (0)