Skip to content

Commit 14c3b83

Browse files
mikewillimikewilli
authored
chore: remove references to old studies platforms and update some for nimbus (#858)
* chore: remove references to old studies platforms and update some for nimbus * lint/spelling
1 parent 25c0c76 commit 14c3b83

File tree

4 files changed

+52
-225
lines changed

4 files changed

+52
-225
lines changed

‎src/cookbooks/client_guidelines.md‎

Lines changed: 1 addition & 159 deletions
Original file line numberDiff line numberDiff line change
@@ -1,161 +1,3 @@
11
# Client Implementation Guidelines for Experiments
22

3-
**Note**: This guidance is useful for implementing Normandy experiments. To ship experiments with the Nimbus platform, please see the guidance for engineers at <https://experimenter.info/>.
4-
5-
There are three supported approaches for enabling experimental features for Firefox:
6-
7-
- [Firefox Prefs](#prefs)
8-
- Prefs can be used to control features that **land in-tree**.
9-
[Feature Gates](#feature-gates) provide a wrapper around prefs that can be used from JavaScript.
10-
- [Firefox Extensions](#extensions) AKA "**Add-ons**".
11-
- If the feature being tested should not land in the tree, or if it will ultimately ship as an extension, then an extension should be used.
12-
13-
New features go through the standard Firefox review, testing, and deployment processes, and are then enabled experimentally in the field using [Normandy][normandy-docs].
14-
15-
## Prefs
16-
17-
Firefox Preferences (AKA "prefs") are commonly used to enable and disable features. However, prefs are more complex to implement correctly than [feature gates](#feature-gates).
18-
19-
**Each pref should represent a different experimental treatment**. If your experimental feature requires multiple prefs, then Normandy does not currently support this but will soon. In the meantime, an [extension](#extensions) such as [multipreffer][multipreffer-docs] may be used.
20-
21-
There are three types of Prefs:
22-
23-
1. Built-in prefs - shipped with Firefox, in `firefox.js`.
24-
2. `user branch` - set by the user, overriding built-in prefs.
25-
3. `default branch` - Overrides both built-in and `user branch` prefs. Only persists until the browser session ends, next restart will revert to either built-in or `user branch` (if set).
26-
27-
[Normandy][normandy-docs] supports overriding both the `user` and `default` branches, although the latter is preferred as it does not permanently override user settings. `default` branch prefs are simple to reset since they do not persist past a restart.
28-
29-
**In order for features to be activated experimentally using `default branch` prefs**:
30-
31-
- The feature must not start up before `final-ui-startup` is observed.
32-
33-
For instance, to set an observer:
34-
35-
```js
36-
Services.obs.addObserver(this, "final-ui-startup", true);
37-
```
38-
39-
In this example, `this` would implement an `observe(subject, topic, data)` function which will be called when `final-ui-startup` is observed. See the [Observer documentation][observer-docs] for more information.
40-
41-
- It must be possible to enable/disable the feature at runtime, via a pref change.
42-
43-
This is similar to the observer pattern above:
44-
45-
```js
46-
Services.prefs.addObserver("pref_name", this);
47-
```
48-
49-
More information is available in the [Preference service documentation][pref-service-docs].
50-
51-
- Never use `Services.prefs.prefHasUserValue()`, or any other function specific to `user branch` prefs.
52-
53-
- Prefs should be set by default in `firefox.js`
54-
55-
If your feature cannot abide by one or more of these rules (for instance, it needs to run at startup and/or cannot be toggled at runtime) then experimental preferences can be set on the `user branch`. This is more complex than using the methods described above; user branch prefs override the users choice, which is a really complex thing to try to support when flipping prefs experimentally. We also need to be careful to back up and reset the pref, and then figure out how to resolve conflicts if the user has changed the pref in the meantime.
56-
57-
## Feature Gates
58-
59-
A new Feature Gate library for Firefox Desktop is now available.
60-
61-
**Each feature gate should represent a different experimental treatment**. If your experimental feature requires multiple flags, then Normandy will not be able to support this directly and an [extension](#extensions) may be used.
62-
63-
### Feature Gate caveats
64-
65-
The current Feature Gate library comes with a few caveats, and may not be appropriate for your situation:
66-
67-
- Only JS is supported.
68-
- Always asynchronous.
69-
70-
Future versions of the Feature Gate API will include C++/Rust support and a synchronous API.
71-
72-
### Using the Feature Gate library
73-
74-
Read [the documentation][feature-gate-docs] to get started.
75-
76-
## Extensions
77-
78-
Firefox currently supports the [Web Extensions API](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions).
79-
80-
**If new WebExtension APIs are needed, they should land in-tree**. Extensions which are signed by Mozilla can load privileged code using the [WebExtension Experiments](https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html), but this is not preferred.
81-
82-
WebExtensions go through the same correctness and performance tests as other features. This is possible using the Mozilla tryserver by dropping your XPI into `testing/profiles/common/extensions` in `mozilla-central` and pushing to Tryserver - see the [Testing Extensions](#testing-extensions) section below.
83-
84-
NOTE - it is ideal to test against the version of Firefox which the extension will release against, but there is a [bug related to artifact builds on release channels][artifact-bug] which must be worked around. The workaround is pretty simple (modify an `artifacts.py` file), but this bug being resolved will make it much simpler.
85-
86-
**Each extension can represent a different experimental treatment (preferred), or the extension can choose the branch internally**.
87-
88-
### SHIELD studies
89-
90-
The previous version of the experiments program, SHIELD, always bundled privileged code with extensions and would do things such as mock UI features in Firefox.
91-
92-
This sort of approach is discouraged for new features - land these (or the necessary WebExtension APIs) in-tree instead.
93-
94-
For the moment, the [SHIELD Study Add-on Utilities](https://github.com/mozilla/shield-studies-addon-utils/) may be used if the extension needs to control the lifecycle of the study, but using one extension per experimental treatment makes this unnecessary and is preferred. The APIs provided by the SHIELD Study Add-on Utilities will be available as privileged APIs shipped with Firefox soon.
95-
96-
# Development and Testing
97-
98-
## Testing Built-in Features
99-
100-
Firefox features go through standard development and testing processes. See the [Firefox developer guide][firefox-dev-docs] for more information.
101-
102-
## Testing Extensions
103-
104-
Extensions do not need to go through the same process, but should take advantage of Mozilla CI and bug tracking systems:
105-
106-
1. Use the Mozilla CI to test changes (tryserver).
107-
2. Performance tests (**this step is required**) - extension XPI files should be placed in `testing/profiles/common/extensions/`, which will cause test harnesses to load the XPI.
108-
3. Custom unit/functional tests (AKA `xpcshell`/`mochitest`) may be placed in `testing/extensions`, although running these tests outside Mozilla CI is acceptable so these are **optional**.
109-
4. Receive reviewer approval. A Firefox peer **must sign off** if this extension contains privileged code, aka WebExtension Experiments.
110-
111-
- Any [Firefox Peer][firefox-peer-list] should be able to do the review, or point you to someone who can.
112-
113-
5. Extension is signed.
114-
6. Email to `[email protected]` is sent to request QA
115-
7. QA approval signed off in Bugzilla.
116-
8. Extension is shipped via [Normandy][normandy-docs].
117-
118-
## Example Extensions Testing Workflow
119-
120-
Note that for the below to work you only need [Mercurial][mercurial] installed, but if you want to do local testing you must be set up to [build Firefox][firefox-build-docs]. You don't need to build Firefox from source; [artifact builds][firefox-artifact-build] are sufficient.
121-
122-
In order to use Mozilla CI (AKA "[Tryserver][try-server-docs]"), you must have a full clone of the `mozilla-central` repository:
123-
124-
```bash
125-
hg clone https://hg.mozilla.org/mozilla-central
126-
cd mozilla-central
127-
```
128-
129-
Copy in unsigned XPI, and commit it to your local Mercurial repo:
130-
131-
```bash
132-
cp ~/src/my-extension.xpi testing/profiles/common/extensions/
133-
hg add testing/profiles/common/extensions/my-extension.xpi
134-
hg commit -m "Bug nnn - Testing my extension" testing/profiles/common/extensions/my-extension.xpi
135-
```
136-
137-
Push to Try:
138-
139-
```bash
140-
./mach try -p linux64,macosx64,win64 -b do -u none -t all --artifact
141-
```
142-
143-
This will run Mozilla CI tests on all platforms
144-
145-
Note that you must have Level 1 commit access to use tryserver. If you are interested in interacting with Mozilla CI from Github (which only requires users to be in the Mozilla GitHub org), check out the [Taskcluster Integration proof-of-concept][taskcluster-integration-poc].
146-
147-
Also note that this requires an investment time to set up just as CircleCI or Travis-CI would, so it's not really appropriate for short-term projects. Use tryserver directly instead.
148-
149-
[feature-gate-docs]: https://firefox-source-docs.mozilla.org/toolkit/components/featuregates/featuregates/index.html
150-
[firefox-dev-docs]: https://firefox-source-docs.mozilla.org/
151-
[mercurial]: https://www.mercurial-scm.org/
152-
[taskcluster-integration-poc]: https://github.com/biancadanforth/taskcluster-integration-poc/
153-
[firefox-build-docs]: https://firefox-source-docs.mozilla.org/setup/index.html
154-
[firefox-artifact-build]: https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html
155-
[try-server-docs]: https://firefox-source-docs.mozilla.org/tools/try/
156-
[observer-docs]: https://searchfox.org/mozilla-central/rev/59e797b66f5ce8a27ede0e7677688931be7aed20/xpcom/ds/nsIObserverService.idl#24-39
157-
[normandy-docs]: https://github.com/mozilla/normandy
158-
[pref-service-docs]: https://searchfox.org/mozilla-central/source/modules/libpref/nsIPrefService.idl
159-
[multipreffer-docs]: https://github.com/nhnt11/multipreffer
160-
[firefox-peer-list]: https://wiki.mozilla.org/Modules/All#Firefox
161-
[artifact-bug]: https://bugzilla.mozilla.org/show_bug.cgi?id=1435403
3+
To ship experiments with the Nimbus platform, please see the guidance for engineers at <https://experimenter.info/getting-started/engineers/for-engineers>.

‎src/cookbooks/normandy_events.md‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,18 @@
11
# Working with Normandy events
22

3-
A common request is to count the number of users who have
4-
enrolled or unenrolled from a SHIELD experiment.
3+
<div class="warning">
4+
Normandy and SHIELD are older experimentation platforms
5+
at Mozilla, but are no longer in use. However, Normandy
6+
events are still in use (as of January 2025) by Jetstream
7+
for computing enrollments and exposures. Note that this
8+
will likely change in the near future because these events
9+
have been ported to Glean and legacy telemetry is being
10+
deprecated across the platform.
11+
12+
For more up to date information on events used by Nimbus,
13+
see <https://experimenter.info/telemetry>.
14+
15+
</div>
516

617
The [`events` table](../datasets/batch_view/events/reference.md)
718
includes Normandy enrollment and unenrollment events
@@ -15,12 +26,6 @@ or name (for add-on experiments).
1526
Normandy events are described in detail in the
1627
[Firefox source tree docs][normandy-doc].
1728

18-
Note that addon studies do not have branch information in the events table,
19-
since addons, not Normandy, are responsible for branch assignment.
20-
For studies built with the obsolete [add-on utilities][`addon-utils`],
21-
branch assignments are published to the
22-
[shield_study] dataset.
23-
2429
## Counting pref-flip enrollment events by branch
2530

2631
The `event_map_values` column of enroll events contains a `branch` key,
@@ -68,5 +73,3 @@ ORDER BY 1, 2, 3
6873
```
6974

7075
[normandy-doc]: https://firefox-source-docs.mozilla.org/toolkit/components/normandy/normandy/data-collection.html#enrollment
71-
[shield_study]: ../datasets/experiment_telemetry.md#telemetryshield_study
72-
[`addon-utils`]: https://github.com/mozilla/shield-studies-addon-utils

‎src/datasets/experiment_telemetry.md‎

Lines changed: 31 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,8 @@
1-
# Analyzing data from SHIELD studies
1+
# Accessing Experiment Telemetry
22

33
This article introduces the datasets that are useful for analyzing studies in Firefox.
4-
After reading this article,
5-
you should understand how to answer questions about
6-
study enrollment,
7-
identify telemetry from clients enrolled in an experiment,
8-
and locate telemetry from add-on studies.
4+
After reading this article, you should understand how to answer questions about
5+
experiment enrollment, and identify telemetry from clients enrolled in an experiment.
96

107
## Table of contents
118

@@ -27,7 +24,34 @@ the slug for add-on experiments is defined in the recipe by a field named `name`
2724

2825
You can find the slug associated with an experiment in Experimenter.
2926

30-
## Tables
27+
## Tables (Glean)
28+
29+
### `experiments` map (Glean)
30+
31+
Glean tables include a `ping_info` column with `experiments` mapping from
32+
experiment slug to a struct containing information about the experiment,
33+
including `branch`.
34+
35+
You can query for enrolled clients using a query like:
36+
37+
```sql
38+
SELECT
39+
-- ... some fields ...,
40+
`mozfun.map.get_key`(ping_info.experiments, '{experiment_slug}').branch
41+
FROM
42+
`moz-fx-data-shared-prod.firefox_desktop.metrics`
43+
WHERE
44+
`mozfun.map.get_key`(ds.ping_info.experiments, '{experiment_slug}') IS NOT NULL
45+
AND DATE(m.submission_timestamp) BETWEEN '<start_date>' AND '<end_date>'
46+
AND normalized_channel = 'release'
47+
-- AND ...other filters...
48+
```
49+
50+
## Tables (Legacy Telemetry)
51+
52+
As of January 2025, legacy telemetry is still used for enrollment and exposure
53+
events. However, while Glean adoption is in progress, these docs remain for
54+
the time being as reference.
3155

3256
### `experiments` map (ping tables)
3357

@@ -85,47 +109,3 @@ The event schema is described
85109
[in the Firefox source tree](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/normandy/lib/TelemetryEvents.sys.mjs).
86110

87111
The `events` table is updated daily.
88-
89-
### `telemetry.shield_study_addon`
90-
91-
The `telemetry.shield_study_addon` table contains SHIELD telemetry from legacy add-on experiments,
92-
i.e. key-value pairs sent with the
93-
`browser.study.sendTelemetry()` method from the
94-
[SHIELD study add-on utilities](https://github.com/mozilla/shield-studies-addon-utils/)
95-
library.
96-
97-
The `study_name` attribute of the `payload` column will contain the identifier
98-
registered with the SHIELD add-on utilities.
99-
This is set by the add-on; sometimes it takes the value of
100-
`applications.gecko.id` from the add-on's `manifest.json`.
101-
This is often not the same as the Normandy slug.
102-
103-
The schema for shield-study-addon pings is described in the
104-
[`mozilla-pipeline-schemas` repository](https://github.com/mozilla-services/mozilla-pipeline-schemas/tree/master/schemas/telemetry/shield-study-addon).
105-
106-
The key-value pairs are present in `data` attribute of the `payload` column.
107-
108-
The `telemetry.shield_study_addon` table contains only full days of data.
109-
If you need access to data with lower latency, you can use the "live" table
110-
`telemetry_live.shield_study_addon_v4` which should have latency significantly
111-
less than 1 hour.
112-
113-
### `telemetry.shield_study`
114-
115-
The `telemetry.shield_study` dataset includes
116-
enrollment and unenrollment events for legacy add-on experiments only,
117-
sent by the [SHIELD study add-on utilities](https://github.com/mozilla/shield-studies-addon-utils/).
118-
119-
The `study_name` attribute of the `payload` column will contain the identifier
120-
registered with the SHIELD add-on utilities.
121-
This is set by the add-on; sometimes it takes the value of
122-
`applications.gecko.id` from the add-on's `manifest.json`.
123-
This is often not the same as the Normandy slug.
124-
125-
Normandy also emits its own enrollment and unenrollment events for these studies,
126-
which are available in the `events` table.
127-
128-
The `telemetry.shield_study` table contains only full days of data.
129-
If you need access to data with lower latency, you can use the "live" table
130-
`telemetry_live.shield_study_v4` which should have latency significantly
131-
less than 1 hour.

‎src/tools/projects.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -80,9 +80,11 @@ See also [`data-docs`][docs] for documentation on datasets.
8080
| [Hardware Report][hwreport_gh] | Firefox Hardware Report, [available here][hwreport] |
8181
| [St. Mocli][stmocli] | A command-line interface to [STMO][stmo] |
8282
| [probe-scraper] | Scrape and publish Telemetry probe data from Firefox |
83-
| [test-tube] | Compare data across branches in experiments |
8483
| [experimenter] | A web application for managing experiments |
85-
| [St. Moab][stmoab] | Automatically generate Redash dashboard for A/B experiments |
84+
| [Jetstream] | Automated analysis for experiments |
85+
| [metric-hub] | Semantic layer for metric definitions |
86+
87+
See also [What Data Tool Should I Use?][data-tools-wiki] for more information on Data Tools and their uses.
8688

8789
[tmo_gh]: https://github.com/mozilla/telemetry-dashboard
8890
[glam]: https://github.com/mozilla/glam
@@ -91,16 +93,16 @@ See also [`data-docs`][docs] for documentation on datasets.
9193
[redashstmo]: https://github.com/mozilla/redash-stmo
9294
[taar]: https://github.com/mozilla/taar
9395
[ensemble]: https://github.com/mozilla/ensemble
94-
[shield]: https://wiki.mozilla.org/index.php?title=Firefox/Shield
9596
[tmo]: https://telemetry.mozilla.org
9697
[stmo]: https://sql.telemetry.mozilla.org
9798
[hwreport_gh]: https://github.com/mozilla/firefox-hardware-report
9899
[hwreport]: https://data.firefox.com/dashboard/hardware
99100
[stmocli]: https://github.com/mozilla/stmocli
100101
[probe-scraper]: https://github.com/mozilla/probe-scraper
101-
[test-tube]: https://github.com/mozilla/firefox-test-tube
102102
[experimenter]: https://github.com/mozilla/experimenter
103-
[stmoab]: https://github.com/mozilla/stmoab
103+
[jetstream]: https://github.com/mozilla/jetstream
104+
[metric-hub]: https://github.com/mozilla/metric-hub
105+
[data-tools-wiki]: https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/375750774/Data+Tools
104106

105107
## Legacy projects
106108

0 commit comments

Comments
 (0)