|
1 | | -# AB Testing |
| 1 | +# Beta AB Test framework |
2 | 2 |
|
3 | | -This directory contains the code and configuration for the AB testing framework used on theguardian.com. |
4 | | - |
5 | | -The [`abTest.ts`](./abTest.ts) module is where you should define your AB tests. |
6 | | - |
7 | | -Add your AB tests to the `abTests` array in the `abTest.ts` file. Each test should have a unique name. |
8 | | - |
9 | | -```ts |
10 | | -{ |
11 | | - name: 'webex-example-test', |
12 | | - description: |
13 | | - 'Test something interesting on the site', |
14 | | - |
15 | | - status: 'ON', |
16 | | - expirationDate: '2050-12-30', |
17 | | - type: 'client', |
18 | | - audienceSize: 10 / 100, |
19 | | - groups: ['control', 'variant'], |
20 | | - shouldForceMetricsCollection: true |
21 | | -} |
22 | | -``` |
23 | | - |
24 | | -When you create a PR that modifies the `abTest.ts` file, a git hook and CI will run checks to ensure that your AB test is valid (not expired, enough space for the test etc.). |
25 | | - |
26 | | -When your PR is merged, the AB test will be automatically deployed to Fastly and be available at the same time as your changes. |
27 | | - |
28 | | -## Guidelines for AB Tests |
29 | | - |
30 | | -### Naming Conventions |
31 | | - |
32 | | -AB tests should be prefixed with the team associated with the test, for example `webex-example-test`. This helps to identify the team responsible for the test and is enforce by typescript validation. |
33 | | - |
34 | | -### Test Size and Groups |
35 | | - |
36 | | -The `audienceSize` is the size of the whole test and is divided between the test groups that you specify. The "resolution" of sizing is down to 0.1%, so groups will be rounded to the nearest 0.1%. |
37 | | - |
38 | | -Convention is to have groups named control and variant, but you can name them as you wish. |
39 | | - |
40 | | -A single group is also possible, for example if you're rolling out a new feature and don't need a control. |
41 | | - |
42 | | -### Client vs Server Side Tests |
43 | | - |
44 | | -All requests are processed by Fastly at the edge, however, ab testing of server-side logic in Frontend or DCR will need to be cached separately. Client side tests do not need to be cached separately, as they are applied in the browser after the response is delivered. |
45 | | - |
46 | | -Ensure that the `type` field is set to either `client` or `server` to indicate the type of test so that server side tests can be cached correctly, and client side tests are not splitting the cache unnecessarily. |
47 | | - |
48 | | -There's a limit of the number of concurrent server-side tests that can be run, enforce by the validation script, so it's important to use client-side tests where possible. |
49 | | - |
50 | | -### Test Expiration |
51 | | - |
52 | | -AB tests should have an expiration date set in the future. This is to ensure that tests do not run indefinitely. |
53 | | - |
54 | | -Expired tests will cause the ab testing validation to fail, and will not be deployed. |
55 | | - |
56 | | -Tests that expire while they are are in-flight will not be served by fastly, and should be removed from the `abTest.ts` file as soon as possible. |
57 | | - |
58 | | -### Audience Spaces |
59 | | - |
60 | | -Ideally AB tests would never overlap (users being in multiple tests), but sometimes this is unavoidable, for example when running a very large 50+% test without interrupting existing tests. |
61 | | - |
62 | | -To add a test where there is not enough space in the default audience space (`A`), you can specify a different `audienceSpace` in the test definition. |
63 | | - |
64 | | -For example if there are already 3 25% tests in space `A` totalling 75%, and you want to run a 50% test, you can set the `audienceSpace` to `B` to allow this test to overlap with the existing tests. |
65 | | - |
66 | | -### Test Status |
67 | | - |
68 | | -Tests can be set to `ON` or `OFF` using the `status` field. Only tests with status `ON` will be validated and deployed. |
| 3 | +This directory contains the code and configuration for the AB testing framework used on theguardian.com. If you're looking to set up a new server or client side AB test using the new framework then please visit the docs [here](https://github.com/guardian/dotcom-rendering/blob/main/dotcom-rendering/docs/development/ab-testing-in-dcr.md#beta-ab-test-framework). |
69 | 4 |
|
70 | 5 | ## How it works |
71 | 6 |
|
|
0 commit comments