Skip to content

Commit f339095

Browse files
JakeiiJakeii
authored
Updates to ab-testing docs and admin dashboard (#14683)
* add opt in/out info to docs * update links in ab-testing dashboard * add opt-in/out instructions to A/B tests overview page * fix test rendering bugs * tweak columns and status colours * couple other nice to haves * cookie -> mvt * use const for margins
1 parent c2e2d9f commit f339095

File tree

8 files changed

+696
-108
lines changed

8 files changed

+696
-108
lines changed

‎ab-testing/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,3 +74,5 @@ The algorithm allocates tests available MVT IDs based on the audience size and s
7474
However, the allocation is completely separate for each audience space, so if you have a test in space `A` and move it to space `B`, it will be allocated different MVT IDs.
7575

7676
The state of the AB tests is stored in Fastly dictionaries, which are updated when the `deploy` task is run. Logic in fastly VCL will then use these dictionaries to determine which users are in which test groups and set appropriate headers and/or cookies.
77+
78+
See the [fastly-edge-cache documentation](https://github.com/guardian/fastly-edge-cache/blob/main/theguardiancom/docs/ab-testing.md) for even more details.

‎ab-testing/frontend/src/lib/components/AudienceBreakdown.svelte‎

Lines changed: 29 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,13 @@
1717
1818
const BAR_HEIGHT = 40;
1919
20-
// Account for legend bar and vertical padding in chart height
21-
const chartHeight = tests.length * BAR_HEIGHT + BAR_HEIGHT + 16;
20+
const BAR_MARGIN_X = 0.1;
21+
const BAR_MARGIN_Y = 2;
2222
2323
const testSpaces = ['A', 'B', 'C'];
2424
25+
const chartHeight = testSpaces.length * BAR_HEIGHT + BAR_HEIGHT + 16;
26+
2527
const testsBySpace = testSpaces.map((space) => {
2628
if (space === 'A') {
2729
return tests.filter(
@@ -33,32 +35,32 @@
3335
});
3436
3537
function getBars(testList: ABTest[], rowPosition: number) {
36-
return testList.reduce<Array<ABTestBarData>>(
37-
(barsList, test, index) => {
38-
const previousBar = barsList.slice(-1)[0];
39-
const offset: number = Number(previousBar?.width ?? 0);
40-
const rowYLevel = index + rowPosition;
41-
const testSize = getSize(test);
42-
43-
return [
44-
...barsList,
45-
{
46-
x: offset,
47-
y: rowYLevel * BAR_HEIGHT + BAR_HEIGHT,
48-
width: testSize,
49-
name: test.name,
50-
segments: `${offset}% to ${offset + testSize}%`,
51-
},
52-
];
53-
},
54-
[],
55-
);
38+
return testList.reduce<Array<ABTestBarData>>((barsList, test) => {
39+
const previousBarsWidth = barsList.reduce(
40+
(acc, bar) => acc + bar.width,
41+
0,
42+
);
43+
const offset: number = Number(previousBarsWidth);
44+
const rowYLevel = rowPosition;
45+
const testSize = getSize(test);
46+
47+
return [
48+
...barsList,
49+
{
50+
x: offset,
51+
y: rowYLevel * BAR_HEIGHT + BAR_HEIGHT,
52+
width: testSize,
53+
name: test.name,
54+
segments: `${offset}% to ${offset + testSize}%`,
55+
},
56+
];
57+
}, []);
5658
}
5759
5860
function getAllRows(testsBySpace: ABTest[][]) {
5961
return testsBySpace.reduce<Array<ABTestBarData>>(
60-
(barsList, testsInSpace) => {
61-
return [...barsList, ...getBars(testsInSpace, barsList.length)];
62+
(barsList, testsInSpace, i) => {
63+
return [...barsList, ...getBars(testsInSpace, i)];
6264
},
6365
[],
6466
);
@@ -86,13 +88,13 @@
8688
</svg>
8789
{#each getAllRows(testsBySpace) as bar}
8890
<svg
89-
x={`${bar.x}%`}
91+
x={`${bar.x + BAR_MARGIN_X}%`}
9092
y={bar.y}
91-
width={`${bar.width}%`}
93+
width={`${bar.width - BAR_MARGIN_X * 2}%`}
9294
height={BAR_HEIGHT}
9395
>
9496
<g class="bar">
95-
<rect height={BAR_HEIGHT} width="100%" rx="4" />
97+
<rect height={BAR_HEIGHT - BAR_MARGIN_Y} width="100%" rx="4" />
9698
<text class="name" x="50%" y="50%">{bar.name}</text>
9799
<text class="segments" x="50%" y="50%">{bar.segments}</text>
98100
</g>

‎ab-testing/frontend/src/lib/components/TableFixed.svelte‎

Lines changed: 51 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,8 @@
1212
function daysToExpiry(expires: string) {
1313
const today = new Date();
1414
const expiresDate = new Date(expires);
15-
const differenceInMilliseconds = expiresDate.getTime() - today.getTime();
15+
const differenceInMilliseconds =
16+
expiresDate.getTime() - today.getTime();
1617
const differenceInDays =
1718
differenceInMilliseconds / (1000 * 60 * 60 * 24);
1819
return Math.floor(differenceInDays);
@@ -21,43 +22,69 @@
2122

2223
<section class="tests">
2324
{#each tests as test}
25+
{@const expired = daysToExpiry(test.expirationDate) < 0}
2426
<table>
27+
<colgroup>
28+
<col span="1" style="width: 25%;" />
29+
<col span="1" style="width: 10%;" />
30+
<col span="1" style="width: 35%;" />
31+
<col span="1" style="width: 10%;" />
32+
<col span="1" style="width: 10%;" />
33+
<col span="1" style="width: 10%;" />
34+
</colgroup>
2535
<thead>
2636
<tr>
2737
<th scope="col">Name</th>
2838
<th scope="col">State</th>
29-
<th scope="col">Variants</th>
39+
<th scope="col">Test Groups</th>
3040
<th scope="col">Expires In</th>
3141
<th scope="col">Audience</th>
32-
<th scope="col">Offset</th>
3342
<th scope="col">Ophan</th>
3443
</tr>
3544
</thead>
3645
<tbody>
3746
<tr>
38-
<th scope="row" class="test-name">{test.name}</th>
39-
<td>{test.status}</td>
47+
<th scope="row" class="test-name"
48+
>{test.name} ({test.type})</th
49+
>
50+
<td
51+
class="status"
52+
class:off={test.status === 'OFF'}
53+
class:expired
54+
>
55+
{#if expired}
56+
EXPIRED
57+
{:else}
58+
{test.status}
59+
{/if}
60+
</td>
4061
<td>
4162
<TestVariants
63+
size={test.audienceSize * 100}
4264
testName={test.name}
4365
testGroups={test.groups}
4466
/>
4567
</td>
46-
<td>{daysToExpiry(test.expirationDate)} days</td>
68+
<td class:expired
69+
>{daysToExpiry(test.expirationDate)} days</td
70+
>
4771
<td>{test.audienceSize * 100}%</td>
48-
<td>0</td>
4972
<td><OphanLink testName={test.name} /></td>
5073
</tr>
5174
<tr>
5275
<th scope="row">Description</th>
53-
<td colspan="6">{test.description}</td>
76+
<td colspan="5">{test.description}</td>
5477
</tr>
5578
</tbody>
5679
</table>
5780
{/each}
5881
</section>
5982

6083
<style>
84+
:root {
85+
--ok-green: #00823b;
86+
--error-red: #d5281b;
87+
}
6188
.tests {
6289
border: 1px solid var(--border-grey);
6390
padding: 8px;
@@ -87,4 +114,20 @@
87114
.test-name {
88115
font-weight: 100;
89116
}
117+
118+
.status {
119+
text-transform: uppercase;
120+
font-weight: bold;
121+
122+
color: var(--ok-green);
123+
124+
&.off {
125+
color: #767676;
126+
}
127+
}
128+
129+
.expired {
130+
color: var(--error-red);
131+
font-weight: bold;
132+
}
90133
</style>

‎ab-testing/frontend/src/lib/components/TestVariants.svelte‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,20 +2,29 @@
22
interface Props {
33
testName: string;
44
testGroups: string[];
5+
size: number;
56
}
67
7-
const { testName, testGroups }: Props = $props();
8+
const { testName, testGroups, size }: Props = $props();
9+
10+
const formatter = new Intl.NumberFormat('en-US', {
11+
style: 'percent',
12+
minimumFractionDigits: 0,
13+
maximumFractionDigits: 2,
14+
});
815
</script>
916

1017
<div>
1118
<ul>
12-
{#each testGroups as group}
19+
{#each testGroups as group, i}
1320
<li>
1421
<a
15-
href={`http://www.theguardian.com/uk#ab-${testName}=${group}`}
16-
>
17-
{group}
18-
</a>
22+
href={`https://www.theguardian.com/ab-tests/opt/in/${testName}:${group}`}
23+
>
24+
{group} ({formatter.format(
25+
((1 / testGroups.length) * size) / 100,
26+
)})
27+
</a>{#if i < testGroups.length - 1}&nbsp;|&nbsp;{/if}
1928
</li>
2029
{/each}
2130
</ul>
@@ -28,7 +37,6 @@
2837
list-style: none;
2938
display: flex;
3039
flex-direction: row;
31-
justify-content: space-between;
3240
flex-wrap: wrap;
3341
}
3442
</style>

‎ab-testing/frontend/src/routes/+page.svelte‎

Lines changed: 25 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,37 @@
22
import { ABTests } from '../../../abTest';
33
import Table from '$lib/components/TableFixed.svelte';
44
import AudienceBreakdown from '$lib/components/AudienceBreakdown.svelte';
5-
6-
const clientSideTests = ABTests.filter((test) => test.type === 'client');
7-
const serverSideTests = ABTests.filter((test) => test.type === 'server');
85
</script>
96

107
<h1 class="headline">A/B Tests</h1>
118
<section>
12-
<p>This page provides an overview of currently running A/B tests on theguardian.com. Please note that the audience segment allocations displayed for non-overlapping tests may not correspond to the actual allocation of MVT IDs, but simply represents how much of the audience is included in each test.</p>
13-
</section>
14-
<section>
15-
<h2 class="sub-headline">Client-side Tests</h2>
16-
<AudienceBreakdown tests={clientSideTests} />
17-
<Table tests={clientSideTests} />
9+
<p>
10+
This page provides an overview of currently running A/B tests on
11+
theguardian.com. Please note that the audience segment allocations
12+
displayed for non-overlapping tests may not correspond to the actual
13+
allocation of MVT IDs, but simply represents how much of the audience is
14+
included in each test.
15+
</p>
16+
<p>
17+
AB tests are defined in <a
18+
href="https://github.com/guardian/dotcom-rendering/blob/main/ab-testing/abTest.ts"
19+
>guardian/dotcom-rendering</a
20+
>
21+
</p>
22+
<p>
23+
Use the test group links in the table to opt in to those test groups,
24+
this will override any cookie based test assignment, and you will only
25+
be in that test until you opt out.
26+
</p>
27+
<p>
28+
<a href="https://www.theguardian.com/ab-tests/opt/out"
29+
>Use this link to opt out of any tests</a
30+
>
31+
</p>
1832
</section>
1933
<section>
20-
<h2 class="sub-headline">Server-side Tests</h2>
21-
<AudienceBreakdown tests={serverSideTests} />
22-
<Table tests={serverSideTests} />
34+
<AudienceBreakdown tests={ABTests} />
35+
<Table tests={ABTests} />
2336
</section>
2437

2538
<style>

‎dotcom-rendering/docs/development/ab-testing-in-dcr.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ When the config is merged, the A/B test will be automatically deployed and be av
1010

1111
Ab test on/off state is controlled only by the config. Expired tests will cause the ab testing validation to fail, they will also not be served. In effect expired tests are turned off "automatically", but their config needs to be cleaned up.
1212

13+
The test will appear in https://frontend.gutools.co.uk/analytics/ab-testing once the config is deployed.
14+
1315
## Putting code changes behind an A/B test (group)
1416

1517
### Use in Components
@@ -62,6 +64,19 @@ The ab test API is also available on the window object as `window.guardian.modul
6264

6365
Server side tests are also available in the CAPI object e.g. `CAPIArticle.config.serverSideABTests`.
6466

67+
## Forcing yourself into a test
68+
69+
Use the opt-in and opt-out URL fragments to force yourself into or out of a test.
70+
71+
When opted-in, the test will override any mvt based assignment and you'll only be in the opted-in test group.
72+
73+
When opted-out, you'll return to random/mvt based assignment.
74+
75+
These links are also in the [frontend admin](https://frontend.gutools.co.uk/analytics/ab-testing).
76+
77+
- Opt-in Example: `https://theguardian.com/ab-tests/opt/in/commercial-test-example:variant`
78+
- Opt-out: `https://theguardian.com/ab-tests/opt/out`
79+
6580
# Legacy A/B testing in DCR
6681

6782
> [!WARNING]

0 commit comments

Comments
 (0)