Skip to content

Commit 244a6b0

Browse files
authored
Expand E2E nigthly test BrowserStack job platforms (#1225)
* Expand E2E nigthly test BrowserStack job platforms * Revert back to Sequoia as Tahoe isn't a wide release yet
1 parent 5201a66 commit 244a6b0

File tree

5 files changed

+147
-32
lines changed

5 files changed

+147
-32
lines changed

.github/workflows/nightly-tests-desktop.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,8 @@ jobs:
4848
build-name: 'TB Appointment Nightly Tests (Desktop): BUILD_INFO'
4949

5050
- name: Run E2E Tests on Production on Browserstack (Desktop)
51+
# don't send GHA failure email if any of the E2E tests fail, can be annoying (I check the jobs each day in BrowserStack)
52+
continue-on-error: true
5153
run: |
5254
cd ./test/e2e
5355
cp .env.prod.example .env

test/e2e/README.md

Lines changed: 16 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -51,24 +51,32 @@ APPT_MY_SHARE_LINK=<apointment-dev-user-share-link>
5151
APPT_BOOKEE_EMAIL=<booking-requesters-email>
5252
```
5353

54-
To run the E2E tests headless (still in `test/e2e`):
54+
To run the E2E tests on Firefox headless (still in `test/e2e`):
5555

5656
```bash
5757
npm run e2e-test
5858
```
5959

60-
To run the E2E tests with a UI so you can watch the tests run (still in `test/e2e`):
60+
To run the E2E tests on Firefox with a UI so you can watch the tests run (still in `test/e2e`):
6161

6262
```bash
6363
npm run e2e-test-headed
6464
```
6565

66-
To run the E2E tests in debug mode (still in `test/e2e`):
66+
To run the E2E tests against google chromium with a UI so you can watch the tests run (still in `test/e2e`):
6767

6868
```bash
69-
npm run e2e-test-debug
69+
npx playwright test --grep e2e-suite --project=chromium --headed
7070
```
7171

72+
To run the E2E tests against safari with a UI so you can watch the tests run (still in `test/e2e`):
73+
74+
```bash
75+
npx playwright test --grep e2e-suite --project=safari --headed
76+
```
77+
78+
Note that for project you can use any of the project/browser names as listed in the [playwright.config.ts](./playwright.config.ts) file (but the browser must be installed on your local machine).
79+
7280
## Running the E2E tests against the staging environmnent
7381

7482
First copy over the provided `.env.stage.example` to a local `.env`:
@@ -87,24 +95,18 @@ APPT_MY_SHARE_LINK=<apointment-stage-user-share-link>
8795
APPT_BOOKEE_EMAIL=<booking-requesters-email>
8896
```
8997

90-
To run the E2E tests headless (still in `test/e2e`):
98+
To run the E2E tests on Firefox headless (still in `test/e2e`):
9199

92100
```bash
93101
npm run e2e-test
94102
```
95103

96-
To run the E2E tests with a UI so you can watch the tests run (still in `test/e2e`):
104+
To run the E2E tests on Firefox with a UI so you can watch the tests run (still in `test/e2e`):
97105

98106
```bash
99107
npm run e2e-test-headed
100108
```
101109

102-
To run the E2E tests in debug mode (still in `test/e2e`):
103-
104-
```bash
105-
npm run e2e-test-debug
106-
```
107-
108110
## Running the production sanity test
109111

110112
First copy over the provided `.env.prod.example` to a local `.env`:
@@ -123,24 +125,18 @@ APPT_MY_SHARE_LINK=<apointment-prod-user-share-link>
123125
APPT_BOOKEE_EMAIL=<booking-requesters-email>
124126
```
125127

126-
To run the production sanity test headless (still in `test/e2e`):
128+
To run the production sanity test on Firefox headless (still in `test/e2e`):
127129

128130
```bash
129131
npm run prod-sanity-test
130132
```
131133

132-
To run the production sanity test with a UI so you can watch the tests run (still in `test/e2e`):
134+
To run the production sanity test on Firefox with a UI so you can watch the tests run (still in `test/e2e`):
133135

134136
```bash
135137
npm run prod-sanity-test-headed
136138
```
137139

138-
To run the production sanity test in debug mode (still in `test/e2e`):
139-
140-
```bash
141-
npm run prod-sanity-test-debug
142-
```
143-
144140
## Running on BrowserStack
145141

146142
You can run the E2E tests from your local machine but against browsers provided in the BrowserStack Automate cloud.
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# =============================
2+
# Set BrowserStack Credentials
3+
# =============================
4+
# In the terminal export BROWSERSTACK_USERNAME=<secret>
5+
# In the terminal export BROWSERSTACK_ACCESS_KEY=<secret>
6+
7+
# ======================
8+
# BrowserStack Reporting
9+
# ======================
10+
# The following capabilities are used to set up reporting on BrowserStack:
11+
# Set 'projectName' to the name of your project. Example, Marketing Website
12+
projectName: Thunderbird Appointment
13+
# Set `buildName` as the name of the job / testsuite being run
14+
buildName: Nightly Appointment E2E Tests (Desktop)
15+
# `buildIdentifier` is a unique id to differentiate every execution that gets appended to
16+
# buildName. Choose your buildIdentifier format from the available expressions:
17+
# ${BUILD_NUMBER} (Default): Generates an incremental counter with every execution
18+
# ${DATE_TIME}: Generates a Timestamp with every execution. Eg. 05-Nov-19:30
19+
# Read more about buildIdentifiers here -> https://www.browserstack.com/docs/automate/selenium/organize-tests
20+
buildIdentifier: '${DATE_TIME}'
21+
22+
# =======================================
23+
# Platforms (Browsers / Devices to test)
24+
# =======================================
25+
# Platforms object contains all the browser / device combinations you want to test on.
26+
# Entire list available here -> (https://www.browserstack.com/list-of-browsers-and-platforms/automate)
27+
platforms:
28+
- os: OS X
29+
osVersion: Sequoia
30+
browserName: playwright-firefox
31+
browserVersion: latest
32+
playwrightConfigOptions:
33+
name: Firefox-OSX
34+
setup:
35+
- name: 'setup_firefox_osx'
36+
testMatch: 'auth.setup.ts'
37+
use:
38+
storageState: 'test-results/.auth/user.json'
39+
40+
- os: OS X
41+
osVersion: Sequoia
42+
browserName: playwright-webkit
43+
browserVersion: 26.0
44+
playwrightConfigOptions:
45+
name: Safari-OSX
46+
setup:
47+
- name: 'setup_safari_osx'
48+
testMatch: 'auth.setup.ts'
49+
use:
50+
storageState: 'test-results/.auth/user.json'
51+
52+
- os: Windows
53+
osVersion: 11
54+
browserName: playwright-chromium
55+
browserVersion: latest
56+
playwrightConfigOptions:
57+
name: Chromium-Win11
58+
setup:
59+
- name: 'setup_chromium_win'
60+
testMatch: 'auth.setup.ts'
61+
use:
62+
storageState: 'test-results/.auth/user.json'
63+
64+
- os: Windows
65+
osVersion: 11
66+
browserName: edge
67+
browserVersion: latest
68+
playwrightConfigOptions:
69+
name: Edge-Win11
70+
setup:
71+
- name: 'setup_edge_win'
72+
testMatch: 'auth.setup.ts'
73+
use:
74+
storageState: 'test-results/.auth/user.json'
75+
76+
# =======================
77+
# Parallels per Platform
78+
# =======================
79+
# The number of parallel threads to be used for each platform set.
80+
# BrowserStack's SDK runner will select the best strategy based on the configured value
81+
#
82+
# Example 1 - If you have configured 3 platforms and set `parallelsPerPlatform` as 2, a total of 6 (2 * 3) parallel threads will be used on BrowserStack
83+
#
84+
# Example 2 - If you have configured 1 platform and set `parallelsPerPlatform` as 5, a total of 5 (1 * 5) parallel threads will be used on BrowserStack
85+
parallelsPerPlatform: 1
86+
87+
# ==========================================
88+
# BrowserStack Local
89+
# (For localhost, staging/private websites)
90+
# ==========================================
91+
# Set browserStackLocal to true if your website under test is not accessible publicly over the internet
92+
# Learn more about how BrowserStack Local works here -> https://www.browserstack.com/docs/automate/selenium/local-testing-introduction
93+
browserstackLocal: false # <boolean> (Default false)
94+
# browserStackLocalOptions:
95+
# Options to be passed to BrowserStack local in-case of advanced configurations
96+
# localIdentifier: # <string> (Default: null) Needed if you need to run multiple instances of local.
97+
# forceLocal: true # <boolean> (Default: false) Set to true if you need to resolve all your traffic via BrowserStack Local tunnel.
98+
# Entire list of arguments available here -> https://www.browserstack.com/docs/automate/selenium/manage-incoming-connections
99+
100+
# ===================
101+
# Debugging features
102+
# ===================
103+
debug: false # <boolean> # Set to true if you need screenshots for every selenium command ran
104+
networkLogs: false # <boolean> Set to true to enable HAR logs capturing; off as may contain sensitive info like login API requests
105+
consoleLogs: info # <string> Remote browser's console debug levels to be printed (`disable`, `errors`, `warnings`, `info`, or `verbose`)
106+
framework: playwright
107+
browserstack.playwrightVersion: 1.55.0 # must match our client playwright version in package.json; this is the latest supported in BrowserStack
108+
browserstack.playwrightLogs: false # disable playwright logs appearing on browserstack builds as may contain sensitive info like credentials
109+
# CUSTOM_TAG_<INT>: # <string> (Default: parent folder name of the test file) Custom tag for your test suite
110+
browserstack.maskBasicAuth: true # mask username and passwords from browserstack session logs
111+
browserstack.maskCommands: sendType, sendPress, setHTTPCredentials, setStorageState, setWebAuthnCredentials # prevent sensitive info appearing in browserstack session logs
112+
# Test Observability is an intelligent test reporting & debugging product. It collects data using the SDK. Read more about what data is collected at https://www.browserstack.com/docs/test-observability/references/terms-and-conditions
113+
# Visit observability.browserstack.com to see your test reports and insights. To disable test observability, specify `testObservability: false` in the key below.
114+
testObservability: false

test/e2e/package.json

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,13 @@
55
"scripts": {
66
"e2e-test": "npx playwright test --grep e2e-suite --project=firefox",
77
"e2e-test-headed": "npx playwright test --grep e2e-suite --project=firefox --headed",
8-
"e2e-test-debug": "npx playwright test --grep e2e-suite --project=firefox --headed --ui",
9-
"e2e-test-browserstack": "npx browserstack-node-sdk playwright test --grep e2e-suite --browserstack.buildName 'Appointment E2E Tests' --browserstack.config 'browserstack-desktop.yml'",
10-
"e2e-test-browserstack-gha": "npx browserstack-node-sdk playwright test --grep e2e-suite --browserstack.config 'browserstack-desktop.yml'",
8+
"e2e-test-browserstack": "npx browserstack-node-sdk playwright test --grep e2e-suite --browserstack.buildName 'Appointment E2E Tests (Desktop)' --browserstack.config 'browserstack-desktop.yml'",
9+
"e2e-test-browserstack-gha": "npx browserstack-node-sdk playwright test --grep e2e-suite --project=Firefox-OSX --browserstack.config 'browserstack-desktop.yml'",
1110
"prod-sanity-test": "npx playwright test --grep prod-sanity --project=firefox",
1211
"prod-sanity-test-headed": "npx playwright test --grep prod-sanity --project=firefox --headed",
13-
"prod-sanity-test-debug": "npx playwright test --grep prod-sanity --project=firefox --headed --ui",
14-
"prod-sanity-test-browserstack": "npx browserstack-node-sdk playwright test --grep prod-sanity --browserstack.buildName 'Production Sanity Test' --browserstack.config 'browserstack-desktop.yml'",
12+
"prod-sanity-test-browserstack": "npx browserstack-node-sdk playwright test --grep prod-sanity --browserstack.buildName 'Production Sanity Test (Desktop)' --browserstack.config 'browserstack-desktop.yml'",
1513
"prod-sanity-test-browserstack-gha": "npx browserstack-node-sdk playwright test --grep prod-sanity --browserstack.config 'browserstack-desktop.yml'",
16-
"prod-nightly-tests-browserstack-gha": "npx browserstack-node-sdk playwright test --grep prod-nightly --browserstack.config 'browserstack-desktop.yml'",
14+
"prod-nightly-tests-browserstack-gha": "npx browserstack-node-sdk playwright test --grep prod-nightly --browserstack.config 'browserstack-desktop-nightly.yml'",
1715
"postinstall": "npm update browserstack-node-sdk"
1816
},
1917
"keywords": [],

test/e2e/playwright.config.ts

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,9 @@ export default defineConfig({
2222
/* Opt out of parallel tests on CI. */
2323
workers: process.env.CI ? 1 : 1, // actualy don't run in parallel locally either, for now
2424
// Global timeout: Playwright will timeout if the entire session (includes all test runs) exceeds this.
25-
// Must take into account running on mulitple browsers (and BrowserStack is much slower too!). Odds are the
25+
// Must take into account total time running on mulitple platforms/browsers in BrowserStack. Odds are the
2626
// tests will time out at the locator/test level first anyway; but there is no default so best to specify
27-
globalTimeout: 12 * 60 * 1000,
27+
globalTimeout: 45 * 60 * 1000,
2828
// Individual test timeout - a single test will time out if it is still running after this time (ms)
2929
timeout: 150 * 1000, // 2.5 minutes
3030
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
@@ -71,10 +71,15 @@ export default defineConfig({
7171
dependencies: ['setup'],
7272
},
7373

74-
//{
75-
// name: 'webkit',
76-
// use: { ...devices['Desktop Safari'] },
77-
//},
74+
{
75+
name: 'safari',
76+
use: {
77+
...devices['Desktop Safari'],
78+
// Use prepared auth state
79+
storageState: 'test-results/.auth/user.json',
80+
},
81+
dependencies: ['setup'],
82+
},
7883

7984
/* Test against mobile viewports. */
8085
// {

0 commit comments

Comments
 (0)