|
| 1 | +# Experiments |
| 2 | + |
| 3 | +There are times when we want make a change, but aren't sure if it will provide a better user experience. |
| 4 | + |
| 5 | +In these scenarios we can run experiments. |
| 6 | + |
| 7 | +Experiments are A/B tests where A is some version of our site and B is another. When the user requests our site they are randomly served either site A or B. |
| 8 | + |
| 9 | +After the experiment is live, we gather data via [events](../../README.md) that help determine which version of the site we want to stick with. |
| 10 | + |
| 11 | +## TOC |
| 12 | + |
| 13 | +- [Experiments as feature flags](#experiments-as-feature-flags) |
| 14 | +- [Experiment variations](#experiment-variations) |
| 15 | +- [Adding an experiment](#adding-an-experiment) |
| 16 | +- [Implementing an experiment](#implementing-an-experiment) |
| 17 | +- [Toggling an experiment on or off](#toggling-an-experiment-on-or-off) |
| 18 | +- [Tracking results on an experiment](#tracking-results-on-an-experiment) |
| 19 | + - [Via regular events](#via-regular-events) |
| 20 | + - [Via the `experiment` event](#via-the-experiment-event) |
| 21 | + |
| 22 | + |
| 23 | +## Experiments as feature flags |
| 24 | + |
| 25 | +An additional benefit of this pattern is that it lets you turn on/off a feature in the UI and toggle it from the developer console. This is useful if you want to ship UI changes in parts, or test something in production before turning it on. |
| 26 | + |
| 27 | +## Experiment variations |
| 28 | + |
| 29 | +To clarify terminology, if a user is shown site A which is the original site _without_ the experiment they will have an `experiment_variation` value of `"control"` to indicate that they are in the control group. |
| 30 | + |
| 31 | +If the user is shown the experiment (site B), they will have an `experiment_variation` value of `"treatment"` |
| 32 | + |
| 33 | +## Adding an experiment |
| 34 | + |
| 35 | +1. Create a `key` for you experiment, e.g. `ai_search_experiment` |
| 36 | +2. Determine how many users will see the experiment. The default is 50% and makes sense for _most_ use cases. |
| 37 | +3. Add your experiment to [experiments.ts](./experiments.ts) |
| 38 | + |
| 39 | +Example, |
| 40 | + |
| 41 | +```typescript |
| 42 | +// Add new experiment key to this list |
| 43 | +export type ExperimentNames = 'example_experiment' | 'ai_search_experiment' |
| 44 | + |
| 45 | +export const EXPERIMENTS = { |
| 46 | + example_experiment: { ... } |
| 47 | + ai_search_experiment: { |
| 48 | + key: 'ai_search_experiment', |
| 49 | + isActive: true, // Set to false when the experiment is over |
| 50 | + percentOfUsersToGetExperiment: 10, // Only 10% of users will get the experiment |
| 51 | + limitToLanguages: ['en'], // Only users with the `en` language will be included in the experiment |
| 52 | + includeVariationInContext: true, // See note below |
| 53 | + }, |
| 54 | +} |
| 55 | +``` |
| 56 | + |
| 57 | +When `includeVariationInContext` is true **all** analytics events sent will include `experiment_variation` in their context. `experiment_variation` will be `"treatment"` or `"control"` depending on which the user was randomly assigned. |
| 58 | + |
| 59 | +> [!IMPORTANT] |
| 60 | +> Since the `experiment_variation` is a single key in the context, **only one experiment** can include their variations in the context e.g. only one value of `includeVariationInContext` can be `true`. |
| 61 | +
|
| 62 | +## Implementing an experiment |
| 63 | + |
| 64 | +For example, let's say you are conducting an experiment that changes the search bar. |
| 65 | + |
| 66 | +In the code that displays the search bar, you can use the `shouldShowExperiment` function to determine which version of the code to show the user. |
| 67 | + |
| 68 | +Example: |
| 69 | + |
| 70 | +```typescript |
| 71 | +import { useRouter } from 'next/router' |
| 72 | +import { useShouldShowExperiment } from '@/events/components/experiments/useShouldShowExperiment' |
| 73 | +import { EXPERIMENTS } from '@/events/components/experiments/experiments' |
| 74 | +import { ClassicSearchBar } from "@/search/components/ClassicSearchBar.tsx" |
| 75 | +import { NewSearchBar } from "@/search/components/NewSearchBar.tsx" |
| 76 | + |
| 77 | +export function SearchBar() { |
| 78 | + const router = useRouter() |
| 79 | + // Users who were randomly placed in the `treatment` group will be shown the experiment |
| 80 | + const { shouldShow: shouldShowNewSearch } = useShouldShowExperiment( |
| 81 | + EXPERIMENTS.ai_search_experiment, |
| 82 | + router.locale |
| 83 | + ) |
| 84 | + |
| 85 | + if (shouldShowNewSearch) { |
| 86 | + return ( |
| 87 | + <NewSearchBar /> |
| 88 | + ) |
| 89 | + } |
| 90 | + return <ClassicSearchBar /> |
| 91 | +} |
| 92 | +``` |
| 93 | + |
| 94 | +> [!NOTE] |
| 95 | +> If a user is placed in the `"treatment"` group e.g. they are shown the experiment and will continue to see the treatment across all sessions from the same browser. This is because we use a hash of user's session ID cookie to deterministically set the control group. The session cookie lasts for 365 days, otherwise they might see something different on each reload. |
| 96 | + |
| 97 | +## Toggling an experiment on or off |
| 98 | + |
| 99 | +In development every session is placed into the `"treatment"` control group so that the experiment can be developed on. |
| 100 | + |
| 101 | +However, you can change which experiment to show by calling the following function in the `Console` tab in Chrome dev tools, |
| 102 | + |
| 103 | +```javascript |
| 104 | +window.overrideControlGroup("<experiment_key>", "treatment" | "control"); |
| 105 | + |
| 106 | +// Example to see original search experience |
| 107 | +window.overrideControlGroup("ai_search_experiment", "control"); |
| 108 | +``` |
| 109 | +
|
| 110 | +For events, you can verify that your `experiment_variation` values are being included in the event context from the `Network` tab in Chrome dev tools. |
| 111 | +
|
| 112 | +## Tracking results on an experiment |
| 113 | +
|
| 114 | +### Via regular events |
| 115 | +
|
| 116 | +If your experiment object in [experiments.ts](./experiments.ts) included the `includeVariationInContext: true` key (and is the ONLY object to include that key) then the `experiment_variation` of your experiment will be sent in the context of an event. |
| 117 | +
|
| 118 | +This means that you can send other events, like |
| 119 | +
|
| 120 | +```typescript |
| 121 | +sendEvent({ |
| 122 | + type: EventType.search, |
| 123 | + search_query: "How do I open pdf?", |
| 124 | + search_context: "general-search", |
| 125 | +}); |
| 126 | +``` |
| 127 | +
|
| 128 | +And the `context` on that event will include the `experiment_variation` key and value of your experiment, |
| 129 | +
|
| 130 | +e.g. |
| 131 | +
|
| 132 | +```javascript |
| 133 | +{ |
| 134 | + search_query: "How do I open pdf?", |
| 135 | + search_context: "general-search", |
| 136 | + context: { |
| 137 | + ... |
| 138 | + experiment_variation: "treatment" // Could also be "control" depending on the random outcome |
| 139 | + } |
| 140 | +} |
| 141 | +``` |
| 142 | +
|
| 143 | +### Via the `experiment` event |
| 144 | +
|
| 145 | +If your experiment is specific, meaning it can be tracked with a boolean event, e.g. |
| 146 | +
|
| 147 | +```javascript |
| 148 | +{ |
| 149 | + experiment_name: <string>, // e.g. `new_button_experiment` for did user click new button? |
| 150 | + experiment_variation: 'treatment' | 'control', |
| 151 | + experiment_success: <boolean>, // e.g. true the user is using the new button! |
| 152 | +} |
| 153 | +``` |
| 154 | +
|
| 155 | +Then you should omit the `includeVariationInContext` key from your experiment object and use the `sendExperimentSuccess` function to track events. |
| 156 | +
|
| 157 | +Example: |
| 158 | +
|
| 159 | +```typescript |
| 160 | +import { sendExperimentSuccess } from '@/events/components/experiments/experiment-event' |
| 161 | +import { EXPERIMENTS } from '@/events/components/experiments/experiments' |
| 162 | + |
| 163 | +export function MyNewComponent() { |
| 164 | + return ( |
| 165 | + <button onClick={() => { |
| 166 | + console.log("The user did the thing!") |
| 167 | + sendExperimentSuccess(EXPERIMENTS.new_button_experiment) |
| 168 | + }}> |
| 169 | + ) |
| 170 | +} |
| 171 | + |
| 172 | +``` |
0 commit comments