Skip to content

Commit a8a29c8

Browse files
Ebonsignoriheiskr
Ebonsignori
and
heiskr
authored
add experiments pattern (#54228)
Co-authored-by: Kevin Heis <[email protected]>
1 parent 7833a26 commit a8a29c8

File tree

15 files changed

+539
-43
lines changed

15 files changed

+539
-43
lines changed

‎src/events/components/events.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import Cookies from 'src/frame/components/lib/cookies'
33
import { parseUserAgent } from './user-agent'
44
import { Router } from 'next/router'
55
import { isLoggedIn } from 'src/frame/components/hooks/useHasAccount'
6+
import { getExperimentVariationForContext } from './experiments/experiment'
67
import { EventType, EventPropsByType } from '../types'
78

89
const COOKIE_NAME = '_docs-events'
@@ -110,6 +111,8 @@ export function sendEvent<T extends EventType>({
110111
color_mode_preference: getColorModePreference(),
111112
os_preference: Cookies.get('osPreferred'),
112113
code_display_preference: Cookies.get('annotate-mode'),
114+
115+
experiment_variation: getExperimentVariationForContext(getMetaContent('path-language')),
113116
},
114117

115118
...props,

‎src/events/components/experiment.ts‎

Lines changed: 0 additions & 34 deletions
This file was deleted.

‎src/events/components/experiments/README.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# Experiments
2+
3+
There are times when we want make a change, but aren't sure if it will provide a better user experience.
4+
5+
In these scenarios we can run experiments.
6+
7+
Experiments are A/B tests where A is some version of our site and B is another. When the user requests our site they are randomly served either site A or B.
8+
9+
After the experiment is live, we gather data via [events](../../README.md) that help determine which version of the site we want to stick with.
10+
11+
## TOC
12+
13+
- [Experiments as feature flags](#experiments-as-feature-flags)
14+
- [Experiment variations](#experiment-variations)
15+
- [Adding an experiment](#adding-an-experiment)
16+
- [Implementing an experiment](#implementing-an-experiment)
17+
- [Toggling an experiment on or off](#toggling-an-experiment-on-or-off)
18+
- [Tracking results on an experiment](#tracking-results-on-an-experiment)
19+
- [Via regular events](#via-regular-events)
20+
- [Via the `experiment` event](#via-the-experiment-event)
21+
22+
23+
## Experiments as feature flags
24+
25+
An additional benefit of this pattern is that it lets you turn on/off a feature in the UI and toggle it from the developer console. This is useful if you want to ship UI changes in parts, or test something in production before turning it on.
26+
27+
## Experiment variations
28+
29+
To clarify terminology, if a user is shown site A which is the original site _without_ the experiment they will have an `experiment_variation` value of `"control"` to indicate that they are in the control group.
30+
31+
If the user is shown the experiment (site B), they will have an `experiment_variation` value of `"treatment"`
32+
33+
## Adding an experiment
34+
35+
1. Create a `key` for you experiment, e.g. `ai_search_experiment`
36+
2. Determine how many users will see the experiment. The default is 50% and makes sense for _most_ use cases.
37+
3. Add your experiment to [experiments.ts](./experiments.ts)
38+
39+
Example,
40+
41+
```typescript
42+
// Add new experiment key to this list
43+
export type ExperimentNames = 'example_experiment' | 'ai_search_experiment'
44+
45+
export const EXPERIMENTS = {
46+
example_experiment: { ... }
47+
ai_search_experiment: {
48+
key: 'ai_search_experiment',
49+
isActive: true, // Set to false when the experiment is over
50+
percentOfUsersToGetExperiment: 10, // Only 10% of users will get the experiment
51+
limitToLanguages: ['en'], // Only users with the `en` language will be included in the experiment
52+
includeVariationInContext: true, // See note below
53+
},
54+
}
55+
```
56+
57+
When `includeVariationInContext` is true **all** analytics events sent will include `experiment_variation` in their context. `experiment_variation` will be `"treatment"` or `"control"` depending on which the user was randomly assigned.
58+
59+
> [!IMPORTANT]
60+
> Since the `experiment_variation` is a single key in the context, **only one experiment** can include their variations in the context e.g. only one value of `includeVariationInContext` can be `true`.
61+
62+
## Implementing an experiment
63+
64+
For example, let's say you are conducting an experiment that changes the search bar.
65+
66+
In the code that displays the search bar, you can use the `shouldShowExperiment` function to determine which version of the code to show the user.
67+
68+
Example:
69+
70+
```typescript
71+
import { useRouter } from 'next/router'
72+
import { useShouldShowExperiment } from '@/events/components/experiments/useShouldShowExperiment'
73+
import { EXPERIMENTS } from '@/events/components/experiments/experiments'
74+
import { ClassicSearchBar } from "@/search/components/ClassicSearchBar.tsx"
75+
import { NewSearchBar } from "@/search/components/NewSearchBar.tsx"
76+
77+
export function SearchBar() {
78+
const router = useRouter()
79+
// Users who were randomly placed in the `treatment` group will be shown the experiment
80+
const { shouldShow: shouldShowNewSearch } = useShouldShowExperiment(
81+
EXPERIMENTS.ai_search_experiment,
82+
router.locale
83+
)
84+
85+
if (shouldShowNewSearch) {
86+
return (
87+
<NewSearchBar />
88+
)
89+
}
90+
return <ClassicSearchBar />
91+
}
92+
```
93+
94+
> [!NOTE]
95+
> If a user is placed in the `"treatment"` group e.g. they are shown the experiment and will continue to see the treatment across all sessions from the same browser. This is because we use a hash of user's session ID cookie to deterministically set the control group. The session cookie lasts for 365 days, otherwise they might see something different on each reload.
96+
97+
## Toggling an experiment on or off
98+
99+
In development every session is placed into the `"treatment"` control group so that the experiment can be developed on.
100+
101+
However, you can change which experiment to show by calling the following function in the `Console` tab in Chrome dev tools,
102+
103+
```javascript
104+
window.overrideControlGroup("<experiment_key>", "treatment" | "control");
105+
106+
// Example to see original search experience
107+
window.overrideControlGroup("ai_search_experiment", "control");
108+
```
109+
110+
For events, you can verify that your `experiment_variation` values are being included in the event context from the `Network` tab in Chrome dev tools.
111+
112+
## Tracking results on an experiment
113+
114+
### Via regular events
115+
116+
If your experiment object in [experiments.ts](./experiments.ts) included the `includeVariationInContext: true` key (and is the ONLY object to include that key) then the `experiment_variation` of your experiment will be sent in the context of an event.
117+
118+
This means that you can send other events, like
119+
120+
```typescript
121+
sendEvent({
122+
type: EventType.search,
123+
search_query: "How do I open pdf?",
124+
search_context: "general-search",
125+
});
126+
```
127+
128+
And the `context` on that event will include the `experiment_variation` key and value of your experiment,
129+
130+
e.g.
131+
132+
```javascript
133+
{
134+
search_query: "How do I open pdf?",
135+
search_context: "general-search",
136+
context: {
137+
...
138+
experiment_variation: "treatment" // Could also be "control" depending on the random outcome
139+
}
140+
}
141+
```
142+
143+
### Via the `experiment` event
144+
145+
If your experiment is specific, meaning it can be tracked with a boolean event, e.g.
146+
147+
```javascript
148+
{
149+
experiment_name: <string>, // e.g. `new_button_experiment` for did user click new button?
150+
experiment_variation: 'treatment' | 'control',
151+
experiment_success: <boolean>, // e.g. true the user is using the new button!
152+
}
153+
```
154+
155+
Then you should omit the `includeVariationInContext` key from your experiment object and use the `sendExperimentSuccess` function to track events.
156+
157+
Example:
158+
159+
```typescript
160+
import { sendExperimentSuccess } from '@/events/components/experiments/experiment-event'
161+
import { EXPERIMENTS } from '@/events/components/experiments/experiments'
162+
163+
export function MyNewComponent() {
164+
return (
165+
<button onClick={() => {
166+
console.log("The user did the thing!")
167+
sendExperimentSuccess(EXPERIMENTS.new_button_experiment)
168+
}}>
169+
)
170+
}
171+
172+
```

‎src/events/components/experiments/experiment-event.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
import { EventType } from '@/events/types'
2+
import { sendEvent } from '../events'
3+
import { getExperimentControlGroupFromSession } from './experiment'
4+
import { ExperimentNames } from './experiments'
5+
6+
export function sendExperimentSuccess(experimentKey: ExperimentNames, success = true) {
7+
return sendEvent({
8+
type: EventType.experiment,
9+
experiment_name: experimentKey,
10+
experiment_variation: getExperimentControlGroupFromSession(experimentKey).toLowerCase(),
11+
experiment_success: success,
12+
})
13+
}

‎src/events/components/experiments/experiment.ts‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
import murmur from 'imurmurhash'
2+
import {
3+
CONTROL_VARIATION,
4+
ExperimentNames,
5+
TREATMENT_VARIATION,
6+
getActiveExperiments,
7+
} from './experiments'
8+
import { getUserEventsId } from '../events'
9+
import Cookies from 'src/frame/components/lib/cookies'
10+
11+
let experimentsInitialized = false
12+
13+
export function shouldShowExperiment(
14+
experimentKey: ExperimentNames | { key: ExperimentNames },
15+
locale: string,
16+
) {
17+
// Accept either EXPERIMENTS.<experiment_key> or EXPERIMENTS.<experiment_key>.key
18+
if (typeof experimentKey === 'object') {
19+
experimentKey = experimentKey.key
20+
}
21+
22+
// Determine if user is in treatment group. If they are, show the experiment
23+
const experiments = getActiveExperiments('all')
24+
for (const experiment of experiments) {
25+
if (experiment.key === experimentKey) {
26+
// If the user has staffonly cookie, and staff override is true, show the experiment
27+
if (experiment.alwaysShowForStaff) {
28+
const staffCookie = Cookies.get('staffonly')
29+
if (staffCookie && staffCookie.startsWith('yes')) {
30+
console.log(`Staff cookie is set, showing '${experiment.key}' experiment`)
31+
return true
32+
}
33+
}
34+
35+
// If there is an override for the current session, use that
36+
if (controlGroupOverride[experiment.key]) {
37+
const controlGroup = getExperimentControlGroupFromSession(
38+
experimentKey,
39+
experiment.percentOfUsersToGetExperiment,
40+
)
41+
return controlGroup === TREATMENT_VARIATION
42+
// Otherwise use the regular logic to determine if the user is in the treatment group
43+
} else if (
44+
experiment.limitToLanguages?.length &&
45+
experiment.limitToLanguages.includes(locale)
46+
) {
47+
return (
48+
getExperimentControlGroupFromSession(
49+
experimentKey,
50+
experiment.percentOfUsersToGetExperiment,
51+
) === TREATMENT_VARIATION
52+
)
53+
}
54+
}
55+
}
56+
return false
57+
}
58+
59+
// Allow developers to override their experiment group for the current session
60+
export const controlGroupOverride = {} as { [key in ExperimentNames]: 'treatment' | 'control' }
61+
if (typeof window !== 'undefined') {
62+
// @ts-expect-error
63+
window.overrideControlGroup = (
64+
experimentKey: ExperimentNames,
65+
controlGroup: 'treatment' | 'control',
66+
): string => {
67+
const activeExperiments = getActiveExperiments('all')
68+
// Make sure key is valid
69+
if (activeExperiments.some((experiment) => experiment.key === experimentKey)) {
70+
controlGroupOverride[experimentKey] = controlGroup
71+
const event = new Event('controlGroupOverrideChanged')
72+
window.dispatchEvent(event)
73+
return `Updated ${experimentKey}. Session is now in the "${controlGroup}" group for this session.`
74+
} else {
75+
throw new Error(
76+
`Invalid experiment key: ${experimentKey}. Must be one of: ${activeExperiments.map((experiment) => experiment.key).join(', ')}`,
77+
)
78+
}
79+
}
80+
}
81+
82+
// Determine if the user is in the treatment or control group for a given experiment
83+
export function getExperimentControlGroupFromSession(
84+
experimentKey: ExperimentNames,
85+
percentToGetExperiment = 50,
86+
) {
87+
if (controlGroupOverride[experimentKey]) {
88+
return controlGroupOverride[experimentKey]
89+
} else if (process.env.NODE_ENV === 'development') {
90+
return TREATMENT_VARIATION
91+
} else if (process.env.NODE_ENV === 'test') {
92+
return CONTROL_VARIATION
93+
}
94+
// We hash the user's events ID to ensure that the user is always in the same group for a given experiment
95+
// This works because the hash is a deterministic and the user's ID is stored in a cookie for 365 days
96+
const id = getUserEventsId()
97+
const hash = murmur(experimentKey).hash(id).result()
98+
const modHash = hash % 100
99+
return modHash < percentToGetExperiment ? TREATMENT_VARIATION : CONTROL_VARIATION
100+
}
101+
102+
export function getExperimentVariationForContext(locale: string) {
103+
const experiments = getActiveExperiments(locale)
104+
for (const experiment of experiments) {
105+
if (experiment.includeVariationInContext) {
106+
return getExperimentControlGroupFromSession(
107+
experiment.key,
108+
experiment.percentOfUsersToGetExperiment,
109+
)
110+
}
111+
}
112+
113+
// When no experiment has `includeVariationInContext: true`
114+
return null
115+
}
116+
117+
export function initializeExperiments(locale: string) {
118+
if (experimentsInitialized) return
119+
experimentsInitialized = true
120+
121+
const experiments = getActiveExperiments(locale)
122+
123+
if (experiments.length && process.env.NODE_ENV === 'development') {
124+
console.log(
125+
`In development, all users are placed in the "${TREATMENT_VARIATION}" group for experiments`,
126+
)
127+
} else if (experiments.length && process.env.NODE_ENV === 'test') {
128+
console.log(`In test, all users are placed in the "${CONTROL_VARIATION}" group for experiments`)
129+
}
130+
131+
let numberOfExperimentsUsingContext = 0
132+
for (const experiment of experiments) {
133+
if (experiment.includeVariationInContext) {
134+
// Validate the experiments object
135+
numberOfExperimentsUsingContext++
136+
if (numberOfExperimentsUsingContext > 1) {
137+
throw new Error(
138+
'Only one experiment can include its variation in the context at a time. Please update the experiments configuration.',
139+
)
140+
}
141+
}
142+
143+
const controlGroup = getExperimentControlGroupFromSession(
144+
experiment.key,
145+
experiment.percentOfUsersToGetExperiment,
146+
)
147+
148+
// Even in preview & prod it is useful to see if a given experiment is "on" or "off"
149+
console.log(
150+
`Experiment ${experiment.key} is in the "${controlGroup === TREATMENT_VARIATION ? TREATMENT_VARIATION : CONTROL_VARIATION}" group for this browser.\nCall function window.overrideControlGroup('${experiment.key}', 'treatment' | 'control') to change your group for this session.`,
151+
)
152+
}
153+
}

0 commit comments

Comments
 (0)