You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: ab-testing/README.md
+1-68Lines changed: 1 addition & 68 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,73 +1,6 @@
1
1
# Beta AB Test framework
2
2
3
-
This directory contains the code and configuration for the new **Beta** AB Test framework, developed by commercial-dev to support both client and server side A/B tests in DCR. The goal is to eventually replace the legacy A/B testing framework with this new framework.
4
-
5
-
Please reach out to commercial-dev if you're setting up a client or server side A/B tests in DCR and you're interested in using this new framework whilst it's in beta (this would be helpful for the team) or if you'd like up to date information on it's state of readiness. **For now the legacy framework is still available to use, and documentation for that can be found [here](https://github.com/guardian/dotcom-rendering/blob/main/dotcom-rendering/docs/development/ab-testing-in-dcr.md)**.
6
-
7
-
The [`abTest.ts`](./abTest.ts) module is where you should define your AB tests.
8
-
9
-
Add your AB tests to the `abTests` array in the `abTest.ts` file. Each test should have a unique name.
10
-
11
-
```ts
12
-
{
13
-
name: 'webex-example-test',
14
-
description:
15
-
'Test something interesting on the site',
16
-
owners: ['webex@guardian.co.uk'],
17
-
status: 'ON',
18
-
expirationDate: '2050-12-30',
19
-
type: 'client',
20
-
audienceSize: 10/100,
21
-
groups: ['control', 'variant'],
22
-
shouldForceMetricsCollection: true
23
-
}
24
-
```
25
-
26
-
When you create a PR that modifies the `abTest.ts` file, a git hook and CI will run checks to ensure that your AB test is valid (not expired, enough space for the test etc.).
27
-
28
-
When your PR is merged, the AB test will be automatically deployed to Fastly and be available at the same time as your changes.
29
-
30
-
## Guidelines for AB Tests
31
-
32
-
### Naming Conventions
33
-
34
-
AB tests should be prefixed with the team associated with the test, for example `webex-example-test`. This helps to identify the team responsible for the test and is enforce by typescript validation.
35
-
36
-
### Test Size and Groups
37
-
38
-
The `audienceSize` is the size of the whole test and is divided between the test groups that you specify. The "resolution" of sizing is down to 0.1%, so groups will be rounded to the nearest 0.1%.
39
-
40
-
Convention is to have groups named control and variant, but you can name them as you wish.
41
-
42
-
A single group is also possible, for example if you're rolling out a new feature and don't need a control.
43
-
44
-
### Client vs Server Side Tests
45
-
46
-
All requests are processed by Fastly at the edge, however, ab testing of server-side logic in Frontend or DCR will need to be cached separately. Client side tests do not need to be cached separately, as they are applied in the browser after the response is delivered.
47
-
48
-
Ensure that the `type` field is set to either `client` or `server` to indicate the type of test so that server side tests can be cached correctly, and client side tests are not splitting the cache unnecessarily.
49
-
50
-
There's a limit of the number of concurrent server-side tests that can be run, enforce by the validation script, so it's important to use client-side tests where possible.
51
-
52
-
### Test Expiration
53
-
54
-
AB tests should have an expiration date set in the future. This is to ensure that tests do not run indefinitely.
55
-
56
-
Expired tests will cause the ab testing validation to fail, and will not be deployed.
57
-
58
-
Tests that expire while they are are in-flight will not be served by fastly, and should be removed from the `abTest.ts` file as soon as possible.
59
-
60
-
### Audience Spaces
61
-
62
-
Ideally AB tests would never overlap (users being in multiple tests), but sometimes this is unavoidable, for example when running a very large 50+% test without interrupting existing tests.
63
-
64
-
To add a test where there is not enough space in the default audience space (`A`), you can specify a different `audienceSpace` in the test definition.
65
-
66
-
For example if there are already 3 25% tests in space `A` totalling 75%, and you want to run a 50% test, you can set the `audienceSpace` to `B` to allow this test to overlap with the existing tests.
67
-
68
-
### Test Status
69
-
70
-
Tests can be set to `ON` or `OFF` using the `status` field. Only tests with status `ON` will be validated and deployed.
3
+
This directory contains the code and configuration for the AB testing framework used on theguardian.com. If you're looking to set up a new server or client side AB test using the new framework then please visit the docs [here](https://github.com/guardian/dotcom-rendering/blob/main/dotcom-rendering/docs/development/ab-testing-in-dcr.md#beta-ab-test-framework).
@@ -78,34 +78,103 @@ You can verify that you have been correctly assigned to the variant by appending
78
78
79
79
You can access server-side `abTests` within DCR wherever the CAPI object is used (`CAPIArticle.config.abTests`).
80
80
81
+
---
82
+
81
83
# Beta AB Test framework
82
84
83
85
This is a new framework that has been developed by commercial-dev to support both client and server side A/B tests in DCR. The goal is to eventually replace the legacy A/B testing framework described above with this new framework.
84
86
85
87
Please get in touch with the commercial-dev team if you'd like up to date information on it's state of readiness.
86
88
89
+
If you're interested in how it works please visit the docs [here](https://github.com/guardian/dotcom-rendering/tree/main/ab-testing#beta-ab-test-framework).
90
+
87
91
## Creating a new A/B test
88
92
89
-
Create an ab test in [ab-testing/abTest.ts](../ab-testing/abTest.ts) both server and client side tests are defined here. More information on defining tests can be found in [ab-testing/README.md](../ab-testing/README.md).
93
+
### 1. Configure your A/B test
94
+
95
+
Create an A/B test in [ab-testing/abTest.ts](../ab-testing/abTest.ts)**both server _and_ client side tests** are defined here.
96
+
97
+
Add your A/B tests to the `abTests` array in the `abTest.ts` file. Each test should have a unique name.
98
+
99
+
```ts
100
+
{
101
+
name: 'webex-example-test',
102
+
description:
103
+
'Test something interesting on the site',
104
+
owners: ['webex@guardian.co.uk'],
105
+
status: 'ON',
106
+
expirationDate: '2050-12-30',
107
+
type: 'client',
108
+
audienceSize: 10/100,
109
+
groups: ['control', 'variant'],
110
+
shouldForceMetricsCollection: true
111
+
}
112
+
```
113
+
114
+
When you create a PR that modifies the `abTest.ts` file, a git hook and CI will run checks to ensure that your A/B test is valid (not expired, enough space for the test etc.).
115
+
116
+
When your PR is merged, the A/B test will be automatically deployed to Fastly and be available at the same time as your changes.
117
+
118
+
#### Guidelines for A/B tests
119
+
120
+
#### Naming Conventions
121
+
122
+
A/B tests should be prefixed with the team associated with the test, for example `webex-example-test`. This helps to identify the team responsible for the test and is enforce by typescript validation, you can inspect & edit the allowed team name definitions [here](https://github.com/guardian/dotcom-rendering/blob/main/ab-testing/types.ts#L9).
123
+
124
+
#### Test Size and Groups
125
+
126
+
The `audienceSize` is the size of the whole test and is divided between the test groups that you specify. The "resolution" of sizing is down to 0.1%, so groups will be rounded to the nearest 0.1%.
127
+
128
+
Convention is to have groups named control and variant, but you can name them as you wish.
129
+
130
+
A single group is also possible, for example if you're rolling out a new feature and don't need a control.
131
+
132
+
#### Client vs Server Side Tests
133
+
134
+
All requests are processed by Fastly at the edge, however, A/B testing of server-side logic in Frontend or DCR will need to be cached separately. Client side tests do not need to be cached separately, as they are applied in the browser after the response is delivered.
135
+
136
+
Ensure that the `type` field is set to either `client` or `server` to indicate the type of test so that server side tests can be cached correctly, and client side tests are not splitting the cache unnecessarily.
137
+
138
+
There's a limit of the number of concurrent server-side tests that can be run, enforce by the validation script, so it's important to use client-side tests where possible.
139
+
140
+
#### Test Expiration
141
+
142
+
A/B tests should have an expiration date set in the future. This is to ensure that tests do not run indefinitely.
143
+
144
+
Expired tests will cause the A/B testing validation to fail, and will not be deployed.
145
+
146
+
Tests that expire while they are are in-flight will not be served by fastly, and should be removed from the `abTest.ts` file as soon as possible.
147
+
148
+
#### Audience Spaces
149
+
150
+
Ideally A/B tests would never overlap (users being in multiple tests), but sometimes this is unavoidable, for example when running a very large 50+% test without interrupting existing tests.
151
+
152
+
To add a test where there is not enough space in the default audience space (`A`), you can specify a different `audienceSpace` in the test definition.
153
+
154
+
For example if there are already 3 25% tests in space `A` totalling 75%, and you want to run a 50% test, you can set the `audienceSpace` to `B` to allow this test to overlap with the existing tests.
155
+
156
+
#### Test Status
157
+
158
+
Tests can be set to `ON` or `OFF` using the `status` field. Only tests with status `ON` will be validated and deployed.
90
159
91
160
When the config is merged, the A/B test will be automatically deployed and be available at the same time as your changes.
92
161
93
-
Ab test on/off state is controlled only by the config. Expired tests will cause the ab testing validation to fail, they will also not be served. In effect expired tests are turned off "automatically", but their config needs to be cleaned up.
162
+
A/B test on/off state is controlled only by the config. Expired tests will cause the A/B testing validation to fail, they will also not be served. In effect expired tests are turned off "automatically", but their config needs to be cleaned up.
94
163
95
164
The test will appear in https://frontend.gutools.co.uk/analytics/ab-testing once the config is deployed.
96
165
97
-
##Putting code changes behind an A/B test (group)
166
+
### 2. Putting your code changes behind an A/B test
98
167
99
-
### Use in Components
168
+
Once your A/B test has been configured you can conditionally put your code changes behind an A/B test participation. The instructions below describe how to do this, and are applicable to both client and server side tests.
100
169
101
-
Again, this applies to both client and server side tests.
The ab test API is also available on the window object as `window.guardian.modules.abTests`, this only works client side. It's best to use the `useBetaAB` hook in react components.
The A/B test API described above is also available on the window object as `window.guardian.modules.abTests`. **Note:** This only works client side, you should use the `useBetaAB` hook described above in React components.
241
+
242
+
#### On the Server
146
243
147
244
Server side tests are also available in the CAPI object e.g. `CAPIArticle.config.serverSideABTests`.
148
245
149
-
## Forcing yourself into a test
246
+
#### In the response headers
247
+
248
+
Fastly sends a user's AB participations via the `x-gu-server-ab-tests` response header (server side A/B tests) and `gu_client_ab_tests` response cookie (client side A/B tests).
249
+
250
+
### 4. Testing your changes on CODE
251
+
252
+
If you want to test your changes on CODE you need to follow these steps:
253
+
254
+
1. Configure the A/B tests on your branch
255
+
256
+
2. Deploy your branch to CODE
257
+
258
+
3. Manually run the [🧪 AB testing CI (CODE)](https://github.com/guardian/dotcom-rendering/actions/workflows/ab-testing-ci-code.yml) worfklow using your branch. This deploys the test config to Fastly CODE.
259
+
260
+
The 3rd step is crucial as Fastly buckets users into tests/cohorts and returns your A/B test participations as response headers.
261
+
262
+
### 5. Forcing yourself into a test on PROD/CODE
150
263
151
264
Use the opt-in and opt-out URL fragments to force yourself into or out of a test.
152
265
@@ -156,5 +269,7 @@ When opted-out, you'll return to random/mvt based assignment.
156
269
157
270
These links are also in the [frontend admin](https://frontend.gutools.co.uk/analytics/ab-testing).
0 commit comments