Skip to content

Commit 308acb0

Browse files
jochen-testingbotjochen-testingbot
committed
improve README
1 parent 56a00dc commit 308acb0

File tree

1 file changed

+257
-35
lines changed

1 file changed

+257
-35
lines changed

‎README.md‎

Lines changed: 257 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ CLI tool to run Espresso, XCUITest and Maestro tests on [TestingBot's](https://t
1010
npm install -g testingbotctl
1111
```
1212

13+
**Requirements:** NodeJS 20 or higher
14+
1315
## Authentication
1416

1517
The CLI requires TestingBot API credentials. You can authenticate in several ways:
@@ -35,110 +37,330 @@ This opens your browser for authentication. After logging in, your credentials a
3537
Run Maestro UI tests on real devices and emulators/simulators.
3638

3739
```sh
38-
testingbot maestro <app> <flows> [options]
40+
testingbot maestro <app> <flows...> [options]
3941
```
4042

4143
**Arguments:**
4244
- `app` - Path to your app file (.apk, .ipa, .app, or .zip)
43-
- `flows` - Path to flow file (.yaml/.yml), directory, .zip, or glob pattern
45+
- `flows` - One or more paths to flow files (.yaml/.yml), directories, .zip files, or glob patterns
4446

45-
**Options:**
47+
**Device Options:**
4648

4749
| Option | Description |
4850
|--------|-------------|
4951
| `--device <name>` | Device name (e.g., "Pixel 9", "iPhone 16") |
5052
| `--platform <name>` | Platform: Android or iOS |
5153
| `--deviceVersion <version>` | OS version (e.g., "14", "17.2") |
52-
| `--orientation <orientation>` | PORTRAIT or LANDSCAPE |
54+
| `--real-device` | Use a real device instead of emulator/simulator |
55+
| `--orientation <orientation>` | Screen orientation: PORTRAIT or LANDSCAPE |
5356
| `--device-locale <locale>` | Device locale (e.g., "en_US", "de_DE") |
54-
| `--timezone <timezone>` | Timezone (e.g., "America/New_York") |
55-
| `--name <name>` | Test name for dashboard |
56-
| `--build <build>` | Build identifier for grouping |
57-
| `--throttle-network <speed>` | Network throttling: 4G, 3G, Edge, airplane, disable |
58-
| `--geo-country-code <code>` | Geographic IP location (ISO code) |
57+
| `--timezone <timezone>` | Timezone (e.g., "America/New_York", "Europe/London") |
58+
59+
**Test Configuration:**
60+
61+
| Option | Description |
62+
|--------|-------------|
63+
| `--name <name>` | Test name for dashboard identification |
64+
| `--build <build>` | Build identifier for grouping test runs |
5965
| `--include-tags <tags>` | Only run flows with these tags (comma-separated) |
6066
| `--exclude-tags <tags>` | Exclude flows with these tags (comma-separated) |
61-
| `-e, --env <KEY=VALUE>` | Environment variable for flows (repeatable) |
62-
| `--maestro-version <version>` | Maestro version to use |
67+
| `-e, --env <KEY=VALUE>` | Environment variable for flows (can be repeated) |
68+
| `--maestro-version <version>` | Maestro version to use (e.g., "2.0.10") |
69+
70+
**Network & Location:**
71+
72+
| Option | Description |
73+
|--------|-------------|
74+
| `--throttle-network <speed>` | Network throttling: 4G, 3G, Edge, airplane, or disable |
75+
| `--geo-country-code <code>` | Geographic IP location (ISO country code, e.g., "US", "DE") |
76+
77+
**Output Options:**
78+
79+
| Option | Description |
80+
|--------|-------------|
6381
| `--async` | Start tests and exit without waiting for results |
6482
| `-q, --quiet` | Suppress progress output |
6583
| `--report <format>` | Download report after completion: html or junit |
66-
| `--report-output-dir <path>` | Directory to save reports |
84+
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
85+
| `--download-artifacts` | Download test artifacts (logs, screenshots, video) |
86+
| `--artifacts-output-dir <path>` | Directory to save artifacts zip (defaults to current directory) |
6787

6888
**Examples:**
6989

7090
```sh
7191
# Basic usage
7292
testingbot maestro app.apk ./flows
7393

94+
# Multiple flow directories
95+
testingbot maestro app.apk ./flows/smoke ./flows/regression ./flows/e2e
96+
7497
# With device selection
7598
testingbot maestro app.apk ./flows --device "Pixel 8" --deviceVersion "14"
7699

77-
# iOS app with tags
78-
testingbot maestro app.ipa ./flows --platform iOS --include-tags "smoke,regression"
100+
# iOS app on real device with tags
101+
testingbot maestro app.ipa ./flows --platform iOS --real-device --include-tags "smoke,regression"
79102

80103
# With environment variables
81104
testingbot maestro app.apk ./flows -e API_URL=https://staging.example.com -e API_KEY=secret
82105

83106
# Download JUnit report
84107
testingbot maestro app.apk ./flows --report junit --report-output-dir ./reports
85108

109+
# Download all artifacts (logs, screenshots, video)
110+
testingbot maestro app.apk ./flows --download-artifacts --build "build-123"
111+
86112
# Run in background (async)
87113
testingbot maestro app.apk ./flows --async
88114
```
89115

116+
---
117+
90118
### Espresso
91119

92120
Run Android Espresso tests on real devices and emulators.
93121

94122
```sh
95-
testingbot espresso [options]
123+
testingbot espresso [appFile] [testAppFile] [options]
96124
```
97125

98-
**Required Options:**
126+
**Arguments:**
127+
- `appFile` - Path to application APK file
128+
- `testAppFile` - Path to test APK file containing Espresso tests
129+
130+
**Device Options:**
99131

100132
| Option | Description |
101133
|--------|-------------|
102-
| `--app <path>` | Path to application under test (.apk) |
103-
| `--test-app <path>` | Path to test application (.apk) |
104-
| `--device <name>` | Real device to use |
105-
| `--emulator <name>` | Emulator to use |
134+
| `--app <path>` | Path to application APK file |
135+
| `--test-app <path>` | Path to test APK file |
136+
| `--device <name>` | Device name (e.g., "Pixel 6", "Samsung.*") |
137+
| `--platform-version <version>` | Android OS version (e.g., "12", "13", "14") |
138+
| `--real-device` | Use a real device instead of an emulator |
139+
| `--tablet-only` | Only allocate tablet devices |
140+
| `--phone-only` | Only allocate phone devices |
141+
| `--locale <locale>` | Device locale (e.g., "en_US", "de_DE") |
142+
| `--timezone <timezone>` | Timezone (e.g., "America/New_York", "Europe/London") |
143+
144+
**Test Configuration:**
106145

107-
**Example:**
146+
| Option | Description |
147+
|--------|-------------|
148+
| `--name <name>` | Test name for dashboard identification |
149+
| `--build <build>` | Build identifier for grouping test runs |
150+
| `--test-runner <runner>` | Custom test instrumentation runner |
151+
| `--language <lang>` | App language (ISO 639-1 code, e.g., "en", "fr", "de") |
152+
153+
**Test Filtering:**
154+
155+
| Option | Description |
156+
|--------|-------------|
157+
| `--class <classes>` | Run tests in specific classes (comma-separated fully qualified names) |
158+
| `--not-class <classes>` | Exclude tests in specific classes |
159+
| `--package <packages>` | Run tests in specific packages (comma-separated) |
160+
| `--not-package <packages>` | Exclude tests in specific packages |
161+
| `--annotation <annotations>` | Run tests with specific annotations (comma-separated) |
162+
| `--not-annotation <annotations>` | Exclude tests with specific annotations |
163+
| `--size <sizes>` | Run tests by size: small, medium, large (comma-separated) |
164+
165+
**Network & Location:**
166+
167+
| Option | Description |
168+
|--------|-------------|
169+
| `--throttle-network <speed>` | Network throttling: 4G, 3G, Edge, or airplane |
170+
| `--geo-location <code>` | Geographic IP location (ISO country code, e.g., "US", "DE") |
171+
172+
**Output Options:**
173+
174+
| Option | Description |
175+
|--------|-------------|
176+
| `--async` | Start tests and exit without waiting for results |
177+
| `-q, --quiet` | Suppress progress output |
178+
| `--report <format>` | Download report after completion: html or junit |
179+
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
180+
181+
**Examples:**
108182

109183
```sh
110-
testingbot espresso \
111-
--app app.apk \
112-
--test-app app-test.apk \
184+
# Basic usage with positional arguments
185+
testingbot espresso app.apk app-test.apk --device "Pixel 8"
186+
187+
# Using named options
188+
testingbot espresso --app app.apk --test-app app-test.apk --device "Pixel 8"
189+
190+
# Real device with specific Android version
191+
testingbot espresso app.apk app-test.apk \
192+
--device "Samsung Galaxy S24" \
193+
--platform-version "14" \
194+
--real-device
195+
196+
# Run specific test classes
197+
testingbot espresso app.apk app-test.apk \
113198
--device "Pixel 8" \
114-
--emulator "Android 14"
199+
--class "com.example.LoginTest,com.example.HomeTest"
200+
201+
# Run tests with annotations
202+
testingbot espresso app.apk app-test.apk \
203+
--device "Pixel 8" \
204+
--annotation "com.example.SmokeTest" \
205+
--size "small,medium"
206+
207+
# With network throttling and geolocation
208+
testingbot espresso app.apk app-test.apk \
209+
--device "Pixel 8" \
210+
--throttle-network "3G" \
211+
--geo-location "DE" \
212+
--language "de"
213+
214+
# Download JUnit report
215+
testingbot espresso app.apk app-test.apk \
216+
--device "Pixel 8" \
217+
--report junit \
218+
--report-output-dir ./reports
115219
```
116220

221+
---
222+
117223
### XCUITest
118224

119225
Run iOS XCUITest tests on real devices and simulators.
120226

121227
```sh
122-
testingbot xcuitest [options]
228+
testingbot xcuitest [appFile] [testAppFile] [options]
123229
```
124230

125-
**Required Options:**
231+
**Arguments:**
232+
- `appFile` - Path to application IPA file
233+
- `testAppFile` - Path to test ZIP file containing XCUITests
234+
235+
**Device Options:**
126236

127237
| Option | Description |
128238
|--------|-------------|
129-
| `--app <path>` | Path to application under test (.ipa or .app) |
130-
| `--test-app <path>` | Path to test application |
131-
| `--device <name>` | Device to use |
239+
| `--app <path>` | Path to application IPA file |
240+
| `--test-app <path>` | Path to test ZIP file |
241+
| `--device <name>` | Device name (e.g., "iPhone 15", "iPad.*") |
242+
| `--platform-version <version>` | iOS version (e.g., "17.0", "18.2") |
243+
| `--real-device` | Use a real device instead of a simulator |
244+
| `--tablet-only` | Only allocate tablet devices |
245+
| `--phone-only` | Only allocate phone devices |
246+
| `--orientation <orientation>` | Screen orientation: PORTRAIT or LANDSCAPE |
247+
| `--locale <locale>` | Device locale (e.g., "DE", "US") |
248+
| `--timezone <timezone>` | Timezone (e.g., "America/New_York", "Europe/London") |
249+
250+
**Test Configuration:**
251+
252+
| Option | Description |
253+
|--------|-------------|
254+
| `--name <name>` | Test name for dashboard identification |
255+
| `--build <build>` | Build identifier for grouping test runs |
256+
| `--language <lang>` | App language (ISO 639-1 code, e.g., "en", "fr", "de") |
257+
258+
**Network & Location:**
259+
260+
| Option | Description |
261+
|--------|-------------|
262+
| `--throttle-network <speed>` | Network throttling: 4G, 3G, Edge, or airplane |
263+
| `--geo-location <code>` | Geographic IP location (ISO country code, e.g., "US", "DE") |
264+
265+
**Output Options:**
266+
267+
| Option | Description |
268+
|--------|-------------|
269+
| `--async` | Start tests and exit without waiting for results |
270+
| `-q, --quiet` | Suppress progress output |
271+
| `--report <format>` | Download report after completion: html or junit |
272+
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
132273

133-
**Example:**
274+
**Examples:**
134275

135276
```sh
136-
testingbot xcuitest \
137-
--app app.ipa \
138-
--test-app app-test.zip \
139-
--device "iPhone 16"
277+
# Basic usage with positional arguments
278+
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16"
279+
280+
# Using named options
281+
testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16"
282+
283+
# Real device with specific iOS version
284+
testingbot xcuitest app.ipa app-test.zip \
285+
--device "iPhone 15 Pro" \
286+
--platform-version "17.2" \
287+
--real-device
288+
289+
# iPad in landscape mode
290+
testingbot xcuitest app.ipa app-test.zip \
291+
--device "iPad Pro" \
292+
--tablet-only \
293+
--orientation LANDSCAPE
294+
295+
# With localization settings
296+
testingbot xcuitest app.ipa app-test.zip \
297+
--device "iPhone 16" \
298+
--locale "DE" \
299+
--language "de" \
300+
--timezone "Europe/Berlin"
301+
302+
# With network throttling and geolocation
303+
testingbot xcuitest app.ipa app-test.zip \
304+
--device "iPhone 16" \
305+
--throttle-network "3G" \
306+
--geo-location "DE"
307+
308+
# Download HTML report
309+
testingbot xcuitest app.ipa app-test.zip \
310+
--device "iPhone 16" \
311+
--report html \
312+
--report-output-dir ./reports
313+
314+
# Run in background
315+
testingbot xcuitest app.ipa app-test.zip \
316+
--device "iPhone 16" \
317+
--async
140318
```
141319

320+
---
321+
322+
## Common Features
323+
324+
### Real-time Progress
325+
326+
By default, the CLI shows real-time progress updates including:
327+
- Test status updates
328+
- Device allocation status
329+
- Live output from Maestro flows
330+
331+
Use `--quiet` to suppress progress output.
332+
333+
### Graceful Shutdown
334+
335+
Press `Ctrl+C` to gracefully stop running tests. The CLI will:
336+
1. Stop all active test runs on TestingBot
337+
2. Clean up resources
338+
3. Exit with appropriate status code
339+
340+
Press `Ctrl+C` twice to force exit immediately.
341+
342+
### Report Downloads
343+
344+
All test frameworks support downloading reports after completion:
345+
346+
```sh
347+
# JUnit XML format (for CI integration)
348+
--report junit --report-output-dir ./reports
349+
350+
# HTML format (for human viewing)
351+
--report html --report-output-dir ./reports
352+
```
353+
354+
### Artifact Downloads (Maestro only)
355+
356+
Download all test artifacts including logs, screenshots, and video recordings:
357+
358+
```sh
359+
testingbot maestro app.apk ./flows --download-artifacts --build "my-build"
360+
```
361+
362+
Artifacts are saved as a zip file named after the `--build` value (or with a timestamp if not provided).
363+
142364
## Exit Codes
143365

144366
- `0` - All tests passed

0 commit comments

Comments
 (0)