Skip to content

Commit 5efad8f

Browse files
[#497]: Add Maestro playbook and informational workflow_dispatch CI (#509)
Document eng/QA journey→flow mapping and residuals, link from CI/QA/onboarding, and add a manual-only Android Maestro Actions workflow (not a merge gate).
1 parent 5573567 commit 5efad8f

7 files changed

Lines changed: 212 additions & 7 deletions

File tree

.github/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Workflows for Fluent Mobile (**Android-only**).
1616
| `preview-build.yml` | PR label `preview-build` | Optional isolated Android preview APK (**PR comment only** — debug) |
1717
| `qa-handoff.yml` | PR merged | Needs QA? Yes → issue handoff + assign Roslin + Project 4 `In QA` |
1818
| `nightly-preview.yml` | cron 23:17 PT (APK) + 09:07 PT (Slack) + `workflow_dispatch` | Nightly **binary-only** Android internal APK (dev API); APK cron identity trusted when GitHub delays; Slack held to 09:00–16:00 PT |
19+
| `maestro-android.yml` | `workflow_dispatch` only | **Informational** Maestro E2E (release APK + emulator). Not a required check. See [docs/guides/maestro.md](../docs/guides/maestro.md) |
1920

2021
## PR template + CODEOWNERS
2122

@@ -69,6 +70,8 @@ Scheduled (and manually dispatchable) workflow [`.github/workflows/nightly-previ
6970
| `EXPO_TOKEN` | EAS CLI auth (required for PR preview + nightly) |
7071
| `PROJECT_BOARD_TOKEN` | Optional PAT for Project 4 Status → `In QA` on merge handoff |
7172
| `SLACK_WEBHOOK_URL` | Incoming webhook for nightly success / failure / skip notices |
73+
| `MAESTRO_EMAIL` / `MAESTRO_PASSWORD` | Optional repo secrets — **required** when dispatching `maestro-android.yml` suite `smokes` |
74+
| `MAESTRO_EMAIL_2` / `MAESTRO_PASSWORD_2` | Optional — **required** for suite `multi-account` only (`harness` needs none) |
7275

7376
Does **not** require the Expo GitHub App — only `EXPO_TOKEN`. Manual run: **Actions → Nightly Preview → Run workflow** (available after this workflow exists on `main`).
7477

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# Informational Android Maestro E2E — NOT a required PR / merge gate (#497).
2+
# Manual only (workflow_dispatch). Do not add schedule until the suite is stable.
3+
# Does not replace Needs QA? / @Roslin22 nightly handoff.
4+
name: Maestro Android (informational)
5+
6+
on:
7+
workflow_dispatch:
8+
inputs:
9+
suite:
10+
description: Maestro suite to run
11+
type: choice
12+
required: true
13+
default: harness
14+
options:
15+
- harness
16+
- smokes
17+
- multi-account
18+
19+
concurrency:
20+
group: maestro-android-${{ github.ref }}-${{ inputs.suite }}
21+
cancel-in-progress: true
22+
23+
permissions:
24+
contents: read
25+
26+
env:
27+
EXPO_PUBLIC_API_BASE_URL: https://dev.api.fluent.bible
28+
# Release APK is not __DEV__; E2E_MODE is for local Debug+Metro only.
29+
ORG_GRADLE_PROJECT_reactNativeArchitectures: x86_64
30+
MAESTRO_VERSION: '2.10.0'
31+
32+
jobs:
33+
maestro-android:
34+
name: Maestro ${{ inputs.suite }}
35+
runs-on: ubuntu-latest
36+
timeout-minutes: 90
37+
38+
steps:
39+
- name: Checkout
40+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
41+
with:
42+
persist-credentials: false
43+
44+
- name: Require secrets for suite
45+
env:
46+
MAESTRO_EMAIL: ${{ secrets.MAESTRO_EMAIL }}
47+
MAESTRO_PASSWORD: ${{ secrets.MAESTRO_PASSWORD }}
48+
MAESTRO_EMAIL_2: ${{ secrets.MAESTRO_EMAIL_2 }}
49+
MAESTRO_PASSWORD_2: ${{ secrets.MAESTRO_PASSWORD_2 }}
50+
run: |
51+
set -euo pipefail
52+
suite="${{ inputs.suite }}"
53+
missing=()
54+
if [[ "${suite}" == "smokes" || "${suite}" == "multi-account" ]]; then
55+
[[ -n "${MAESTRO_EMAIL:-}" ]] || missing+=("MAESTRO_EMAIL")
56+
[[ -n "${MAESTRO_PASSWORD:-}" ]] || missing+=("MAESTRO_PASSWORD")
57+
fi
58+
if [[ "${suite}" == "multi-account" ]]; then
59+
[[ -n "${MAESTRO_EMAIL_2:-}" ]] || missing+=("MAESTRO_EMAIL_2")
60+
[[ -n "${MAESTRO_PASSWORD_2:-}" ]] || missing+=("MAESTRO_PASSWORD_2")
61+
fi
62+
if ((${#missing[@]} > 0)); then
63+
echo "::error::Missing repository secrets for suite=${suite}: ${missing[*]}"
64+
echo "Configure secrets listed in docs/guides/maestro.md / .github/README.md (#497)."
65+
exit 1
66+
fi
67+
echo "Secrets OK for suite=${suite} (harness needs none)."
68+
69+
- name: Setup Node
70+
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
71+
with:
72+
node-version-file: '.nvmrc'
73+
cache: 'npm'
74+
75+
- name: Setup Java 17
76+
uses: actions/setup-java@dded0888837ed1f317902acf8a20df0ad188d165 # v5.0.0
77+
with:
78+
distribution: temurin
79+
java-version: '17'
80+
81+
- name: Enable KVM
82+
run: |
83+
set -euo pipefail
84+
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
85+
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
86+
sudo udevadm control --reload-rules
87+
sudo udevadm trigger --name-match=kvm
88+
89+
- name: Install npm dependencies
90+
run: npm ci
91+
92+
- name: Install Maestro CLI
93+
run: npm run maestro:install
94+
95+
- name: Prebuild Android (CNG)
96+
run: npm run prebuild
97+
env:
98+
CI: 'true'
99+
100+
# Assemble inside emulator-runner so ANDROID_HOME / SDK from the action are available.
101+
- name: Build APK + run Maestro on emulator
102+
uses: reactivecircus/android-emulator-runner@1dcd0090116d15e7c562f8db72807de5e036a4ed # v2.34.0
103+
env:
104+
MAESTRO_EMAIL: ${{ secrets.MAESTRO_EMAIL }}
105+
MAESTRO_PASSWORD: ${{ secrets.MAESTRO_PASSWORD }}
106+
MAESTRO_EMAIL_2: ${{ secrets.MAESTRO_EMAIL_2 }}
107+
MAESTRO_PASSWORD_2: ${{ secrets.MAESTRO_PASSWORD_2 }}
108+
with:
109+
api-level: 34
110+
target: google_apis
111+
arch: x86_64
112+
profile: pixel_6
113+
disable-animations: true
114+
emulator-options: -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim
115+
emulator-boot-timeout: 900
116+
script: |
117+
set -euo pipefail
118+
export PATH="${HOME}/.maestro/bin:${PATH}"
119+
chmod +x android/gradlew
120+
(
121+
cd android
122+
./gradlew :app:assembleRelease --no-daemon -PreactNativeArchitectures=x86_64
123+
)
124+
APK="android/app/build/outputs/apk/release/app-release.apk"
125+
test -f "${APK}"
126+
adb install -r "${APK}"
127+
mkdir -p .maestro/test_output
128+
suite="${{ inputs.suite }}"
129+
case "${suite}" in
130+
harness)
131+
npm run maestro:test:harness -- --format junit --output .maestro/test_output/report.xml
132+
;;
133+
smokes)
134+
npm run maestro:test:smokes -- --format junit --output .maestro/test_output/report.xml
135+
;;
136+
multi-account)
137+
npm run maestro:test:multi-account -- --format junit --output .maestro/test_output/report.xml
138+
;;
139+
*)
140+
echo "Unknown suite: ${suite}" >&2
141+
exit 1
142+
;;
143+
esac
144+
adb logcat -d > .maestro/test_output/logcat.txt || true
145+
146+
- name: Upload Maestro artifacts
147+
if: always()
148+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
149+
with:
150+
name: maestro-${{ inputs.suite }}-${{ github.run_id }}
151+
path: |
152+
.maestro/test_output
153+
android/app/build/outputs/apk/release/app-release.apk
154+
if-no-files-found: warn
155+
retention-days: 14

docs/AGENT_ONBOARDING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,7 +205,7 @@ Keep changes **small and scoped** — avoid drive-by refactors.
205205
- **Expo mocks:** [`src/test/mocks/`](../src/test/mocks/) — global `moduleNameMapper` in `jest.config.cjs` for `expo-secure-store`, `expo-file-system`, `expo-audio`.
206206
- **Colocated:** `src/**/*.test.ts(x)` — e.g. `src/utils/logger.test.ts`, `src/services/recordingSync.test.ts`.
207207
- **Live API test:** `fluent-api.test.ts` is **skipped by default**; opt in with `RUN_LIVE_API_TESTS=1 npm test -- fluent-api.test.ts`.
208-
- **Maestro (opt-in Android E2E):** harness + domain smokes — see [guides/maestro.md](./guides/maestro.md). Not a PR/CI merge gate.
208+
- **Maestro (opt-in Android E2E):** harness + domain smokes + multi-account — see [guides/maestro.md](./guides/maestro.md). Playbook maps QA journeys → flows; residual manual listed there. Informational Actions: `maestro-android.yml` (`workflow_dispatch` only). **Not** a PR/CI merge gate; does not replace Needs QA? / nightly handoff ([guides/qa-process.md](./guides/qa-process.md)).
209209
- **Gap:** `src/db/` has partial unit coverage — prefer adding tests when changing queries/repository.
210210

211211
When adding features: mock `op-sqlite`, navigation, and sync in screen tests following existing patterns. Reset shared Expo mocks in `beforeEach` when mutating secure-store/file-system state.

docs/ci.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ This repo runs GitHub Actions on pushes and pull requests. This doc maps what ru
1616
| `qa-handoff.yml` | Post-merge ticket handoff | **Needs QA? Yes** → comment + assign `@Roslin22` + `In QA`; **Needs QA? No**`Done` + close linked issues ([guides/qa-process.md](guides/qa-process.md)) |
1717
| `nightly-preview.yml` | Nightly Android APK | 23:17 PT APK cron (trusted even when GitHub delays) + 09:07 PT Slack (09:00–16:00 PT); `workflow_dispatch`; install comments on recent handoffs |
1818
| `eas-build.yml` | Tag → version sync | Production release path on `v*` tags |
19+
| `maestro-android.yml` | Maestro Android (informational) | **Manual only** (`workflow_dispatch`): build release APK + emulator + Maestro suite. **Not** a PR/merge gate. Secrets: `MAESTRO_EMAIL` / `MAESTRO_PASSWORD` (smokes); + `_2` pair (multi-account). See [guides/maestro.md](guides/maestro.md) |
1920

2021
Local mirrors (run before claiming PR-ready):
2122

@@ -114,6 +115,7 @@ PR template for the GitHub UI: [`.github/PULL_REQUEST_TEMPLATE.md`](../.github/P
114115
- Nightly: `nightly-preview.yml` + [`.github/scripts/nightly-notify-qa-issues.cjs`](../.github/scripts/nightly-notify-qa-issues.cjs) posts install URL on recent handoff issues
115116
- **Process:** [guides/qa-process.md](guides/qa-process.md) (Needs QA?, post-merge nightly QA)
116117
- Human install steps: [guides/qa-preview-testing.md](guides/qa-preview-testing.md)
118+
- Opt-in Maestro E2E (not a merge gate): [guides/maestro.md](guides/maestro.md) + informational `maestro-android.yml`
117119
- **Merge rule:** engineer approval + required CI — QA does **not** block merge
118120
- Production: tag `v*``eas-build.yml` + [`.eas/README.md`](../.eas/README.md)
119121

docs/guides/maestro.md

Lines changed: 48 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,59 @@
11
# Maestro (Android E2E)
22

3-
Opt-in **Android-only** Maestro suite for Fluent Mobile ([#488](https://github.com/eten-tech-foundation/fluent-mobile/issues/488)): harness ([#489](https://github.com/eten-tech-foundation/fluent-mobile/issues/489)) + domain smokes ([#491](https://github.com/eten-tech-foundation/fluent-mobile/issues/491)) + multi-account isolation ([#495](https://github.com/eten-tech-foundation/fluent-mobile/issues/495)).
3+
Opt-in **Android-only** Maestro suite for Fluent Mobile ([#488](https://github.com/eten-tech-foundation/fluent-mobile/issues/488)): harness ([#489](https://github.com/eten-tech-foundation/fluent-mobile/issues/489)) + domain smokes ([#491](https://github.com/eten-tech-foundation/fluent-mobile/issues/491)) + multi-account isolation ([#495](https://github.com/eten-tech-foundation/fluent-mobile/issues/495)) + informational CI playbook ([#497](https://github.com/eten-tech-foundation/fluent-mobile/issues/497)).
44

5-
**Not** a `/create-pr` or required PR merge gate. CI playbook stays in a later ticket ([#497](https://github.com/eten-tech-foundation/fluent-mobile/issues/497)).
5+
**Not** a `/create-pr` or required PR merge gate. Manual Actions: [`.github/workflows/maestro-android.yml`](../../.github/workflows/maestro-android.yml) (`workflow_dispatch` only). Needs QA? / `@Roslin22` nightly handoff is unchanged.
6+
7+
## Playbook — eng vs QA
8+
9+
| Audience | Use Maestro for | Still do manually |
10+
| --- | --- | --- |
11+
| **Engineering** | Local Debug + Metro (`maestro:android:up`, suite scripts) while building auth/nav/record/sync/account features; optional informational Actions dispatch | Exploratory UX, hardware mics/cameras, store builds, anything not encoded below |
12+
| **QA** | Optional: same flows against a Debug client when debugging a Maestro failure | **Nightly APK** checklist and device QA per [qa-process.md](qa-process.md) — Maestro does **not** replace `@Roslin22` handoff |
13+
14+
### Journey → flow map
15+
16+
| Human / QA journey | Maestro | Command / tag |
17+
| --- | --- | --- |
18+
| Cold launch → login screen | `flows/smoke-launch.yaml` | `npm run maestro:test:harness` |
19+
| Sign in / session restore / logout | `flows/android/smoke-auth.yaml` | `npm run maestro:test:auth` |
20+
| Home → My Work / Projects → drafting tabs | `flows/android/smoke-nav.yaml` | `npm run maestro:test:nav` |
21+
| Record → stop → play → delete take | `flows/android/smoke-record.yaml` | `npm run maestro:test:record` |
22+
| Sync Now + Prepare for Offline chrome | `flows/android/smoke-sync-offline.yaml` | `npm run maestro:test:sync` |
23+
| Forgot password / legal / permission deny | `flows/android/smoke-edges.yaml` | `npm run maestro:test:edges` |
24+
| Multi-account nightly A–D | `flows/android/smoke-multi-account.yaml` | `npm run maestro:test:multi-account` |
25+
| All single-account smokes | tag `smoke` | `npm run maestro:test:smokes` |
26+
27+
### Residual manual (not automated)
28+
29+
- Exploratory / visual polish on the **nightly** APK
30+
- Hardware mic quality, Bluetooth, low-storage, OEM permission variants
31+
- Multi-account **A3** (3-account cap) and **E** (sign-out when another account remains) — see [qa-multi-account-nightly.md](qa-multi-account-nightly.md)
32+
- Forced reauth mid-session (needs a backend/session hook)
33+
- Production / Play Store builds (never set `EXPO_PUBLIC_E2E_MODE` on EAS production profiles)
34+
- Raising `MAX_DEVICE_ACCOUNTS` / account-cap product changes
35+
36+
## Informational CI (`workflow_dispatch`)
37+
38+
[`.github/workflows/maestro-android.yml`](../../.github/workflows/maestro-android.yml) builds a local Android **release** APK (CNG prebuild + Gradle; **no Metro**, no `EXPO_PUBLIC_E2E_MODE`), boots an API 34 emulator, and runs one suite. **Not** required on PRs; **no** cron until stable. Day-to-day eng iteration remains Debug + Metro (below).
39+
40+
| Input `suite` | Secrets required |
41+
| --- | --- |
42+
| `harness` | none |
43+
| `smokes` | `MAESTRO_EMAIL`, `MAESTRO_PASSWORD` |
44+
| `multi-account` | both pairs (`MAESTRO_EMAIL_2`, `MAESTRO_PASSWORD_2` too) |
45+
46+
Dispatch: **Actions → Maestro Android (informational) → Run workflow** (after the workflow exists on the target branch / `main`). Artifacts: JUnit + Maestro output + APK (14-day retention).
47+
48+
See also [docs/ci.md](../ci.md) and [`.github/README.md`](../../.github/README.md).
649

750
## Prerequisites
851

952
- JDK **17+** (`JAVA_HOME`)
1053
- Android SDK `platform-tools` (`adb`)
1154
- Emulator or physical device (USB debugging)
12-
- **Debug / expo-dev-client APK** — not Expo Go, not store builds
55+
- **Local eng playbook:** Debug / expo-dev-client APK + Metro — not Expo Go, not store builds
56+
- **Informational CI:** release APK produced in the workflow (no Metro / no `EXPO_PUBLIC_E2E_MODE`)
1357
- Dedicated **Maestro translator** on `dev.api.fluent.bible` with **≥1 chapter assignment** (My Work + project data for nav/record/sync smokes)
1458
- For multi-account: a **second** dedicated translator (`MAESTRO_EMAIL_2` / `MAESTRO_PASSWORD_2`), also with ≥1 assignment — prefer a **different** first My Work display label than account A (isolation assert C1–C3)
1559

@@ -151,4 +195,4 @@ Prefer Maestro `id:` matching React Native `testID`.
151195

152196
## Out of scope
153197

154-
Informational CI / playbook ([#497](https://github.com/eten-tech-foundation/fluent-mobile/issues/497)), auth bypass, iOS, merge-gating Maestro, raising the 3-account device cap.
198+
Auth bypass, iOS, merge-gating Maestro, raising the 3-account device cap, replacing Needs QA? nightly handoff.

docs/guides/qa-process.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# QA process (merge → nightly → pass/fail)
22

3-
Canonical **when / who / board handoff** for Fluent Mobile QA. For install steps on a phone, see [qa-preview-testing.md](qa-preview-testing.md).
3+
Canonical **when / who / board handoff** for Fluent Mobile QA. For install steps on a phone, see [qa-preview-testing.md](qa-preview-testing.md). Opt-in Maestro E2E (engineering / debug; not a merge gate): [maestro.md](maestro.md). Multi-account nightly checklist: [qa-multi-account-nightly.md](qa-multi-account-nightly.md).
44

55
**Invariant:** Engineer approval + green required CI is enough to **merge**. Device QA is **post-merge** on the next **nightly** Android APK (main). QA no longer blocks merge.
66

scripts/maestro-test-multi-account.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,5 @@ fi
3030
exec bash "${ROOT}/scripts/maestro-test.sh" \
3131
--config .maestro/config.yaml \
3232
--include-tags multi-account \
33-
.maestro
33+
.maestro \
34+
"$@"

0 commit comments

Comments
 (0)