Skip to content
This repository was archived by the owner on Jul 9, 2025. It is now read-only.

Commit 6cede0c

Browse files
author
Tarik Eshaq
committed
Bug 1890784: Adds documentation for TPS tests. r=sync-reviewers,markh,lina
Differential Revision: https://phabricator.services.mozilla.com/D207188
1 parent 0978f82 commit 6cede0c

File tree

3 files changed

+352
-0
lines changed

3 files changed

+352
-0
lines changed

‎docs/config.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ categories:
6767
- testing/webrender
6868
- testing/mochitest-plain
6969
- testing/xpcshell
70+
- testing/tps
7071
- web-platform
7172
- gtest
7273
- tools/fuzzing

‎testing/tps/docs/index.md

Lines changed: 349 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,349 @@
1+
# TPS
2+
## Table of Contents
3+
- [What is TPS](#what-is-tps)
4+
- [Why TPS Exists](#why-tps-exists)
5+
- [Architecture](#architecture)
6+
- [Test Format](#test-format)
7+
- [Writing new TPS tests](#writing-new-tps-tests)
8+
- [How to run TPS](#how-to-run-tps)
9+
10+
The majority of this document is targeting the Firefox developer that would like to understand TPS and/or write TPS tests. If you'd like to simply run TPS, see the [How to run TPS](#how-to-run-tps) section.
11+
12+
## What is TPS
13+
TPS is a test automation framework for Firefox Sync. TPS takes in test configuration as JavaScript files defined in [a configuration file](#test-format) and runs all tests in sequence.
14+
15+
Each TPS test is made of one or more [phases](#phases) that each have their own assertions and run in sequence. If **any** phase fails, then the test fails and the remaining phases are not run. However, other tests will still run.
16+
17+
## Why TPS exists
18+
TPS runs against the real Mozilla Accounts server and Sync servers and thus is a good way to test Firefox Sync end-to-end without mocking the servers.
19+
20+
## Architecture
21+
### High level Diagram
22+
The following diagram describes the flow when running TPS.
23+
24+
```{mermaid}
25+
flowchart TD
26+
TPS[TPS Runner]-->T[Run Test Suite]
27+
T-->T1[Run Next Single Test]
28+
T1-->S[Setup]
29+
S-->P[Run Next Phase]
30+
P-->LP[Launch Test Profile]
31+
LP-->Q{Phase Success?}
32+
Q-->|No| CL[Clean up]
33+
Q-->|Yes| Q2{Any more phases?}
34+
Q2-->|Yes|P
35+
Q2-->|No| CL
36+
CL-->CR[Collect Results]
37+
CR-->Q3{Any more tests?}
38+
Q3-->|Yes|T1
39+
Q3-->|No| D[Done]
40+
```
41+
42+
### Single Test Sequence Diagram
43+
The following sequence diagram describes the involved entities executing a single TPS test
44+
```{mermaid}
45+
sequenceDiagram
46+
actor U as User
47+
participant TPS as TPS Runner
48+
participant F as Firefox Profile
49+
participant TE as TPS Extension
50+
U->>TPS: Start test
51+
TPS->>TPS: Parse Test file
52+
loop Every Phase
53+
TPS->>F: Launch Phase w/prefs to install TPS extension
54+
F->>TE: Install Extension
55+
TE->>F: Read prefs to get test file
56+
F->>TE: Test file and TPS Runner configuration
57+
loop Every instruction
58+
TE->>F: Execute test instructions and assertions
59+
F->>TE: Result
60+
end
61+
TE->>F: Phase Done
62+
F->>TPS: Done
63+
TPS->>F: Read logs to get test results
64+
end
65+
TPS->>U: Print test results
66+
```
67+
68+
69+
### Phases
70+
Each Phase is mapped to a Firefox profile, phases may re-use profiles if they are mapped to them, see [the phases object](#the-phases-object) for details on the mapping. All phases are signed in to the same Mozilla Account.
71+
#### Example
72+
For example, a simple test that defines two phases, one that uploads bookmarks and another that downloads them can be described as follows, see the section on [Test Format](#test-format) for details on the format of the tests.
73+
```js
74+
EnableEngines(["bookmarks"]); // Enables the bookmark engine across all profiles(phases) that this test runs
75+
76+
/*
77+
* The list of phases mapped to their corresponding profiles. The object
78+
* here must be in JSON format as it will get parsed by the Python
79+
* testrunner. It is parsed by the YAML package, so it relatively flexible.
80+
*/
81+
var phases = {
82+
phase1: "profile1",
83+
phase2: "profile2",
84+
};
85+
86+
// the initial list of bookmarks to add to the browser
87+
var bookmarksInitial = {
88+
menu: [
89+
{ folder: "foldera" },
90+
{ folder: "folderb" },
91+
],
92+
93+
"menu/foldera": [{ uri: "http://www.cnn.com", title: "CNN" }], // One bookmark in menu/foldera pointing to CNN
94+
"menu/folderb": [{ uri: "http://www.apple.com", title: "Apple"}], // One bookmark in menu/folderb pointing to apple
95+
};
96+
97+
// Add bookmarks to profile1 and sync.
98+
Phase("phase1", [
99+
[Bookmarks.add, bookmarksInitial],
100+
[Bookmarks.verify, bookmarksInitial],
101+
[Sync],
102+
[Bookmarks.verify, bookmarksInitial],
103+
]);
104+
105+
106+
// Sync, then verify that bookmarks added by phase 1 are present.
107+
Phase("phase2", [
108+
[Sync],
109+
[Bookmarks.verify, bookmarksInitial],
110+
]);
111+
```
112+
113+
### The TPS Extension
114+
When the Firefox profile representing the phase loads, it first installs an [extension](https://searchfox.org/mozilla-central/source/services/sync/tps/extensions/tps). The extension is what executes the tests by instructing Firefox and reading from Firefox to assert that Sync is working properly.
115+
116+
The test files execute in the extension, and the [extension defines all the functions that the test files may use](https://searchfox.org/mozilla-central/source/services/sync/tps/extensions/tps/resource/tps.sys.mjs). For instance, in the above [example](#example) the `Phase` function is defined [here](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tps/extensions/tps/resource/tps.sys.mjs#1234)
117+
118+
## Test Format
119+
### Test group configuration
120+
The tests are referenced by a `json` file that references all the tests TPS will run. By default TPS will run the configuration in [`services/sync/tests/tps/all_tests.json`](https://searchfox.org/mozilla-central/source/services/sync/tests/tps/all_tests.json).
121+
The test group configuration is a `json` object with one key named `tests` that is itself a `json` object. The `tests` object has a key for every test to run, and the key should be the name of the [test file](#test-files). The value for each test file is the empty object `{}` if the test should run, or
122+
`{"disabled": "Bug <BUG NUMBER>"}` where `<BUG NUMBER>` is a Bugzilla bug number referencing why the test was disabled.
123+
124+
125+
### Test Files
126+
#### The phases object
127+
Test Files are JavaScript files that will be run by the TPS extension once the Firefox profile is loaded. However, before that is done the TPS framework will load the first object defined in test file as `yaml`. In other words, for the following example:
128+
```js
129+
var phases = {
130+
phase1: "profile1",
131+
phase2: "profile2",
132+
phase3: "profile1", // phase 1 and 3 reuse the same profile, "profile1"
133+
}
134+
135+
// ... rest of the test file
136+
```
137+
The inside of the curly brackets will be parsed as yaml to identify the profiles that TPS will run and map each phase to a profile. In the example above, `phase1` and `phase3` reuse the same profile, but `phase2` uses it's own profile. The rest of the file is regular JavaScript that will be loaded in the [TPS Extension](#the-tps-extension).
138+
#### Test File Capabilities
139+
The TPS Extension exports a set of functions and objects that test files may use. See the [`tps.sys.mjs`](https://searchfox.org/mozilla-central/source/services/sync/tps/extensions/tps/resource/tps.sys.mjs) for up-to-date details, the following is the list of export capabilities as of April 10th, 2024:
140+
##### Enabling Sync Engines
141+
To enable sync engines, which you should do before executing a test that depends on an engine, use the `EnableEngines` function. The function takes an array of strings representing the engine names. Eg:
142+
```js
143+
EnableEngines(["bookmarks", "history"]);
144+
```
145+
##### Start a phase
146+
Phases are run on their assigned profiles profiles in the order they are declared in [the phases object](#the-phases-object). To define what a phase does use the `Phase` function that takes in a phase name as the first argument - **this should be the same name defined in the [phases object](#the-phases-object)** - and a 2D array of actions as a second argument. Each action array in the 2D is run in sequence. The inner array defines what the action is, and any arguments to that action. For example:
147+
```js
148+
Phase("phase1", [[Sync], [Bookmarks.add, bookmarkInitial]]);
149+
```
150+
151+
Note that the example assumes that the phases object and `bookmarkInitial` are defined in test file.
152+
153+
In the above example:
154+
- `"phase1"` is the name of the phase, which should match exactly the phase name as defined in [the phases object](#the-phases-object).
155+
- The 2D array passed as the second argument has:
156+
- `[Sync]` as its first array which means that the phase will first Sync
157+
- `[Bookmarks.add, bookmarkInitial]` as its second member which means that after the Sync, a Bookmarks.add operation will run, and it will add the `bookmarkInitial`.
158+
159+
##### Actions
160+
Actions are run inside phases as arguments to the [`Phase`](#start-a-phase) function.
161+
###### General Actions
162+
There are few general actions that aren't tied to a data type:
163+
- `Sync`: Will trigger a sync
164+
- `Login`: Logs in to Mozilla Account. **You shouldn't need to do this in most cases as it is done automatically**
165+
- `WipeServer`: Will wipe all data from the server
166+
- `EnsureTracking`: Will wait until sync tracking has started. **You shouldn't need to do this in most cases**
167+
168+
###### Data Type specific Actions
169+
170+
Some actions are supported for individual data types. Those actions are triggered by using `<DataType>.<Action>`. The following is a list of all possible `<DataType>` values:
171+
For more details on which actions are supported for which data type, see the following sections.
172+
- `Bookmarks`
173+
- `Tabs`
174+
- `Formdata`
175+
- `History`
176+
- `Passwords`
177+
- `Addons`
178+
- `Addresses`
179+
- `CreditCards`
180+
- `ExtStorage`
181+
- `Prefs`
182+
- `Windows`
183+
184+
**Bookmarks Actions**
185+
186+
187+
Example in [test_existing_bookmarks.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_existing_bookmarks.js)
188+
- `add`: To add a bookmark tree
189+
- `modify`: To modify the existing bookmark tree
190+
- `delete`: To remove a bookmark node from the tree
191+
- `verify`: To verify the bookmark tree matches exactly the tree given, otherwise fails the phase
192+
- `verifyNot`: The inverse of verify, fails the phase if the bookmark tree matches the given tree
193+
- `skipValidation`: To tell Firefox to stop validating bookmark trees.
194+
195+
196+
**Addresses Actions**
197+
198+
199+
Example in [test_addresses.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_addresses.js)
200+
- `add`: Adds an array of addresses
201+
- `modify`: Modifies the list of addresses to what was given
202+
- `delete`: Deletes the given addresses
203+
- `verify`: Verifies that the addresses match exactly the given list, otherwise fails the phase
204+
- `verifyNot`: The inverse of verify, fails the phase if the list of addresses matches the given list.
205+
206+
**CreditCard Actions**
207+
208+
209+
Example in [test_creditcards.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_creditcards.js)
210+
- `add`: Adds an array of credit cards
211+
- `modify`: Modifies the list of credit cards to what was given
212+
- `delete`: Deletes the given credit cards
213+
- `verify`: Verifies that the credit cards match exactly the given list, otherwise fails the phase
214+
- `verifyNot`: The inverse of verify, fails the phase if the list of credit cards matches the given list.
215+
216+
**Addons Actions**
217+
218+
219+
Example in [test_addon_reconciling.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_addon_reconciling.js)
220+
- `installs`: installs a list of addons
221+
- `setEnabled`: Enables or disables a list of addons
222+
- `uninstall`: Uninstalls a list of addons
223+
- `verify`: Verifies that the addons match exactly the given list, otherwise fails the phase
224+
- `verifyNot`: The inverse of verify, fails the phase if the list of addons matches the given list.
225+
- `skipValidation`: Tells Firefox to stop validating Addons.
226+
227+
**Formdata Actions**
228+
229+
230+
Example in [test_formdata.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_formdata.js)
231+
- `add`: Adds an array of form data
232+
- `delete`: Deletes the given form data
233+
- `verify`: Verifies that form data match exactly the given list, otherwise fails the phase
234+
- `verifyNot`: The inverse of verify, fails the phase if the form data matches the given list.
235+
236+
**History Actions**
237+
238+
239+
Example in [test_history.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_history.js)
240+
- `add`: Adds an array of history items
241+
- `delete`: Deletes the given history items
242+
- `verify`: Verifies that form data match exactly the given list, otherwise fails the phase
243+
- `verifyNot`: The inverse of verify, fails the phase if the form data matches the given list.
244+
245+
**Passwords Actions**
246+
247+
248+
Example in [test_passwords.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_passwords.js)
249+
- `add`: To add a list of logins
250+
- `modify`: To modify the existing list of logins based on changes in the given list
251+
- `delete`: To remove a list of logins
252+
- `verify`: To verify the list of logins matches exactly the list given, otherwise fails the phase
253+
- `verifyNot`: The inverse of verify, fails the phase if the logins matches the given list
254+
- `skipValidation`: To tell Firefox to stop validating logins
255+
256+
**Prefs Actions**
257+
258+
259+
Example in [test_prefs.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_prefs.js)
260+
- `modify`: To modify the existing prefs based on changes in the given list
261+
- `verify`: To verify the values of the given prefs match the values given, otherwise fails the phase
262+
263+
**Tabs Actions**
264+
265+
266+
Example in [test_tabs.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_tabs.js)
267+
- `add`: To add a new list of remote tabs, each annotated with which profile it belongs to
268+
- `verify`: To verify the values of remote tabs matches the given list, otherwise fails the phase
269+
270+
**Windows Actions**
271+
272+
273+
Example in [test_privbrw_tabs.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_privbrw_tabs.js#79-84)
274+
- `add`: Adds a new window with the given configuration
275+
276+
**ExtStorage Actions**
277+
278+
279+
Example in [test_extstorage.js](https://searchfox.org/mozilla-central/rev/1f27a4022f9f1269d897526c1c892a57743e650c/services/sync/tests/tps/test_extstorage.js)
280+
281+
- `set`: Sets the value of given key to the given value
282+
- `verify`: Verifies that the value of the given key is the given value, fails the phase otherwise.
283+
284+
## Writing new TPS tests
285+
To write new TPS tests follow the following instructions:
286+
1. Create a new `.js` file in `services/sync/tests/tps` named `test_<my_test_name>.js`
287+
2. Link to the file in the `services/sync/tests/tps/all_tests.json`
288+
3. Follow the format described in [Test Files](#test-files)
289+
4. Make sure to test the file by [running TPS](#how-to-run-tps)
290+
291+
## How to run TPS
292+
### Installation
293+
TPS requires several packages to operate properly. To install TPS and
294+
required packages, use the create_venv.py script provided in the `testing/tps` directory:
295+
```sh
296+
python3 create_venv.py /path/to/create/virtualenv
297+
```
298+
299+
This script will create a virtalenv and install TPS into it.
300+
301+
You must then activate the virtualenv by executing:
302+
303+
- (mac/linux):
304+
```sh
305+
source /path/to/virtualenv/Scripts/activate
306+
```
307+
- (win):
308+
```sh
309+
/path/to/virtualenv/Scripts/activate.bat
310+
```
311+
312+
TPS can then be run by executing:
313+
```sh
314+
runtps --binary=/path/to/firefox
315+
```
316+
317+
> Note: You can run the tps tests in headless mode by using `MOZ_HEADLESS=1`. This will make
318+
> your computer somewhat useable while the tests are running.
319+
320+
When you are done with TPS, you can deactivate the virtualenv by executing
321+
`deactivate`
322+
323+
### Configuration
324+
To edit the TPS configuration, do not edit config/config.json.in in the tree.
325+
Instead, edit config.json inside your virtualenv; it will be located at the
326+
top level of where you specified the virtualenv be created - eg, for the
327+
example above, it will be `/path/to/create/virtualenv/config.json`
328+
329+
#### Setting Up Test Accounts
330+
331+
##### Mozilla Accounts
332+
To create a test account for using the Mozilla Account authentication perform the
333+
following steps:
334+
335+
> Note: Currently, the TPS tests rely on how restmail returns the verification code
336+
> You should use restmail or something very similar.
337+
> Gmail and other providers might give a `The request was blocked for security reasons`
338+
339+
1. Go to the URL: http://restmail.net/mail/%account_prefix%@restmail.net
340+
- Replace `%account_prefix%` with your own test name
341+
2. Go to https://accounts.firefox.com/signup?service=sync&context=fx_desktop_v1
342+
3. Sign in with the previous chosen email address and a password
343+
4. Go back to the Restmail URL, reload the page
344+
5. Search for the verification link and open that page
345+
346+
Now you will be able to use this account for TPS. Note that these
347+
steps can be done in either a test profile or in a private browsing window - you
348+
might want to avoid doing that in a "real" profile that's already connected to
349+
Sync.

‎testing/tps/moz.build

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,7 @@
44
# License, v. 2.0. If a copy of the MPL was not distributed with this
55
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
66

7+
SPHINX_TREES["/testing/tps"] = "docs"
8+
79
with Files("**"):
810
BUG_COMPONENT = ("Testing", "General")

0 commit comments

Comments
 (0)