Skip to content

Commit e6de4df

Browse files
quanruquanru
authored
test(computer,report): add e2e tests for extension playground and recorder (#2195)
* test(computer,report): add e2e tests for extension and report - Add playground advanced tests (aiQuery, aiAssert, multi-todo) - Add recorder mode tests (session, start/stop, code gen) - Add report YAML e2e tests (sidebar, player, detail, theme) - Add parallel CI jobs for new test files * fix(computer): simplify e2e tests and fix recorder flow - Reduce playground test steps to avoid timeout - Fix recorder test to match auto-start behavior - Remove report-e2e CI job (demo.html not available) * test(computer): combine test cases and add settings/cross-mode tests - Combine playground tests: aiQuery+aiAssert+fail in one it() - Combine recorder tests: full lifecycle in one it() - Add chrome-extension-settings.test.ts: settings modal, 3-mode rotation - Add parallel CI job for settings tests (6 total jobs) * fix(computer): fix bridge port conflict and split playground test - Bridge: kill port before starting, wait for LISTENING, async kill - Playground: split aiQuery into own it(), increase sleep to 30s * fix(computer): increase timeout, fix bridge UI wait, simplify recorder - Increase testTimeout to 480s for playground and recorder - Add 3s sleep after bridge connect for UI update - Split recorder lifecycle into 2 shorter tests - Remove overly complex delete session test * fix(computer): increase settings timeout, merge recorder close step - Settings: increase timeout to 480s, remove extra bridge assert - Recorder: merge close+return into lifecycle test * fix(computer): stabilize playground and recorder tests - Playground: separate tests, remove flaky aiAssert fail scenario - Recorder: remove fragile REC state assertion * fix(computer): increase recorder timeout and bridge wait time - Recorder: increase testTimeout to 600s for long lifecycle test - Bridge: increase post-connect wait to 5s for UI update * fix(computer): remove flaky switch-back step from recorder test The mode switch back to Playground is not core recorder logic and causes intermittent timeout failures. * fix(report): remove duplicate player-controls e2e test player-autoplay.yaml and player-settings.yaml from main already cover play/pause and subtitle toggle. The duplicate was flaky. * fix(computer): remove deprecated OPENAI env vars from CI and helpers OPENAI_API_KEY, OPENAI_BASE_URL, MIDSCENE_OPENAI_INIT_CONFIG_JSON are no longer used by the chrome extension. Use MIDSCENE_* vars only. * fix(report): fix sidebar-navigation test assertions Simplify assertions to match actual report UI. The sidebar shows task execution rows, not column headers by default.
1 parent 3ec881d commit e6de4df

File tree

10 files changed

+752
-26
lines changed

10 files changed

+752
-26
lines changed

‎.github/workflows/headless-linux.yml‎

Lines changed: 222 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -121,9 +121,6 @@ jobs:
121121
MIDSCENE_COMPUTER_HEADLESS_LINUX: "true"
122122
CI: "1"
123123
# Env vars to inject into Chrome extension's config UI via CDP
124-
MIDSCENE_OPENAI_INIT_CONFIG_JSON: ${{ secrets.MIDSCENE_OPENAI_INIT_CONFIG_JSON }}
125-
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
126-
OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }}
127124
MIDSCENE_USE_QWEN3_VL: "1"
128125

129126
steps:
@@ -199,9 +196,6 @@ jobs:
199196
MIDSCENE_REPORT_QUIET: "true"
200197
MIDSCENE_COMPUTER_HEADLESS_LINUX: "true"
201198
CI: "1"
202-
MIDSCENE_OPENAI_INIT_CONFIG_JSON: ${{ secrets.MIDSCENE_OPENAI_INIT_CONFIG_JSON }}
203-
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
204-
OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }}
205199
MIDSCENE_USE_QWEN3_VL: "1"
206200

207201
steps:
@@ -263,3 +257,225 @@ jobs:
263257
- name: Check test result
264258
if: steps.test-bridge.outcome == 'failure'
265259
run: exit 1
260+
261+
chrome-extension-playground:
262+
runs-on: ubuntu-22.04
263+
env:
264+
MIDSCENE_MODEL_API_KEY: ${{ secrets.MIDSCENE_MODEL_API_KEY }}
265+
MIDSCENE_MODEL_BASE_URL: ${{ vars.MIDSCENE_MODEL_BASE_URL }}
266+
MIDSCENE_MODEL_NAME: ${{ vars.MIDSCENE_MODEL_NAME }}
267+
MIDSCENE_MODEL_FAMILY: ${{ vars.MIDSCENE_MODEL_FAMILY }}
268+
MIDSCENE_MODEL_RETRY_COUNT: "2"
269+
MIDSCENE_MODEL_RETRY_INTERVAL: "60000"
270+
MIDSCENE_REPORT_QUIET: "true"
271+
MIDSCENE_COMPUTER_HEADLESS_LINUX: "true"
272+
CI: "1"
273+
MIDSCENE_USE_QWEN3_VL: "1"
274+
275+
steps:
276+
- uses: actions/checkout@v4
277+
with:
278+
ref: ${{ github.event.inputs.branch || github.ref }}
279+
280+
- name: Setup pnpm
281+
uses: pnpm/action-setup@v2
282+
with:
283+
version: 9.3.0
284+
285+
- name: Setup Node.js
286+
uses: actions/setup-node@v4
287+
with:
288+
node-version: '24.13.0'
289+
cache: 'pnpm'
290+
291+
- name: Install system dependencies
292+
run: |
293+
sudo apt-get update
294+
sudo apt-get install -y \
295+
xvfb \
296+
x11-xserver-utils \
297+
imagemagick \
298+
libxtst6 \
299+
libxinerama1 \
300+
libx11-6 \
301+
libxkbcommon-x11-0 \
302+
libpng16-16 \
303+
libnss3 \
304+
libatk-bridge2.0-0 \
305+
libdrm2 \
306+
libgbm1 \
307+
libasound2
308+
wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
309+
sudo dpkg -i google-chrome-stable_current_amd64.deb || sudo apt-get -f install -y
310+
311+
- name: Install dependencies
312+
run: pnpm install --frozen-lockfile
313+
314+
- name: Build project
315+
run: pnpm run build:skip-cache
316+
317+
- name: Run Playground advanced tests
318+
run: AI_TEST_TYPE=computer npx nx test @midscene/computer -- tests/ai/chrome-extension-playground.test.ts
319+
timeout-minutes: 40
320+
id: test-playground
321+
continue-on-error: true
322+
323+
- name: Upload test report
324+
if: always()
325+
uses: actions/upload-artifact@v4
326+
with:
327+
name: chrome-extension-playground-report
328+
path: packages/computer/midscene_run/report
329+
if-no-files-found: ignore
330+
331+
- name: Check test result
332+
if: steps.test-playground.outcome == 'failure'
333+
run: exit 1
334+
335+
chrome-extension-recorder:
336+
runs-on: ubuntu-22.04
337+
env:
338+
MIDSCENE_MODEL_API_KEY: ${{ secrets.MIDSCENE_MODEL_API_KEY }}
339+
MIDSCENE_MODEL_BASE_URL: ${{ vars.MIDSCENE_MODEL_BASE_URL }}
340+
MIDSCENE_MODEL_NAME: ${{ vars.MIDSCENE_MODEL_NAME }}
341+
MIDSCENE_MODEL_FAMILY: ${{ vars.MIDSCENE_MODEL_FAMILY }}
342+
MIDSCENE_MODEL_RETRY_COUNT: "2"
343+
MIDSCENE_MODEL_RETRY_INTERVAL: "60000"
344+
MIDSCENE_REPORT_QUIET: "true"
345+
MIDSCENE_COMPUTER_HEADLESS_LINUX: "true"
346+
CI: "1"
347+
MIDSCENE_USE_QWEN3_VL: "1"
348+
349+
steps:
350+
- uses: actions/checkout@v4
351+
with:
352+
ref: ${{ github.event.inputs.branch || github.ref }}
353+
354+
- name: Setup pnpm
355+
uses: pnpm/action-setup@v2
356+
with:
357+
version: 9.3.0
358+
359+
- name: Setup Node.js
360+
uses: actions/setup-node@v4
361+
with:
362+
node-version: '24.13.0'
363+
cache: 'pnpm'
364+
365+
- name: Install system dependencies
366+
run: |
367+
sudo apt-get update
368+
sudo apt-get install -y \
369+
xvfb \
370+
x11-xserver-utils \
371+
imagemagick \
372+
libxtst6 \
373+
libxinerama1 \
374+
libx11-6 \
375+
libxkbcommon-x11-0 \
376+
libpng16-16 \
377+
libnss3 \
378+
libatk-bridge2.0-0 \
379+
libdrm2 \
380+
libgbm1 \
381+
libasound2
382+
wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
383+
sudo dpkg -i google-chrome-stable_current_amd64.deb || sudo apt-get -f install -y
384+
385+
- name: Install dependencies
386+
run: pnpm install --frozen-lockfile
387+
388+
- name: Build project
389+
run: pnpm run build:skip-cache
390+
391+
- name: Run Recorder mode tests
392+
run: AI_TEST_TYPE=computer npx nx test @midscene/computer -- tests/ai/chrome-extension-recorder.test.ts
393+
timeout-minutes: 40
394+
id: test-recorder
395+
continue-on-error: true
396+
397+
- name: Upload test report
398+
if: always()
399+
uses: actions/upload-artifact@v4
400+
with:
401+
name: chrome-extension-recorder-report
402+
path: packages/computer/midscene_run/report
403+
if-no-files-found: ignore
404+
405+
- name: Check test result
406+
if: steps.test-recorder.outcome == 'failure'
407+
run: exit 1
408+
409+
chrome-extension-settings:
410+
runs-on: ubuntu-22.04
411+
env:
412+
MIDSCENE_MODEL_API_KEY: ${{ secrets.MIDSCENE_MODEL_API_KEY }}
413+
MIDSCENE_MODEL_BASE_URL: ${{ vars.MIDSCENE_MODEL_BASE_URL }}
414+
MIDSCENE_MODEL_NAME: ${{ vars.MIDSCENE_MODEL_NAME }}
415+
MIDSCENE_MODEL_FAMILY: ${{ vars.MIDSCENE_MODEL_FAMILY }}
416+
MIDSCENE_MODEL_RETRY_COUNT: "2"
417+
MIDSCENE_MODEL_RETRY_INTERVAL: "60000"
418+
MIDSCENE_REPORT_QUIET: "true"
419+
MIDSCENE_COMPUTER_HEADLESS_LINUX: "true"
420+
CI: "1"
421+
MIDSCENE_USE_QWEN3_VL: "1"
422+
423+
steps:
424+
- uses: actions/checkout@v4
425+
with:
426+
ref: ${{ github.event.inputs.branch || github.ref }}
427+
428+
- name: Setup pnpm
429+
uses: pnpm/action-setup@v2
430+
with:
431+
version: 9.3.0
432+
433+
- name: Setup Node.js
434+
uses: actions/setup-node@v4
435+
with:
436+
node-version: '24.13.0'
437+
cache: 'pnpm'
438+
439+
- name: Install system dependencies
440+
run: |
441+
sudo apt-get update
442+
sudo apt-get install -y \
443+
xvfb \
444+
x11-xserver-utils \
445+
imagemagick \
446+
libxtst6 \
447+
libxinerama1 \
448+
libx11-6 \
449+
libxkbcommon-x11-0 \
450+
libpng16-16 \
451+
libnss3 \
452+
libatk-bridge2.0-0 \
453+
libdrm2 \
454+
libgbm1 \
455+
libasound2
456+
wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
457+
sudo dpkg -i google-chrome-stable_current_amd64.deb || sudo apt-get -f install -y
458+
459+
- name: Install dependencies
460+
run: pnpm install --frozen-lockfile
461+
462+
- name: Build project
463+
run: pnpm run build:skip-cache
464+
465+
- name: Run Settings and cross-mode tests
466+
run: AI_TEST_TYPE=computer npx nx test @midscene/computer -- tests/ai/chrome-extension-settings.test.ts
467+
timeout-minutes: 30
468+
id: test-settings
469+
continue-on-error: true
470+
471+
- name: Upload test report
472+
if: always()
473+
uses: actions/upload-artifact@v4
474+
with:
475+
name: chrome-extension-settings-report
476+
path: packages/computer/midscene_run/report
477+
if-no-files-found: ignore
478+
479+
- name: Check test result
480+
if: steps.test-settings.outcome == 'failure'
481+
run: exit 1

‎apps/report/e2e/detail-panel.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
target:
2+
serve: ./dist
3+
url: demo.html
4+
tasks:
5+
- name: view screenshots tab
6+
flow:
7+
- sleep: 2000
8+
- ai: Click on the first task row in the left sidebar that has type "Planning" or "Insight" or "Locate"
9+
- sleep: 2000
10+
- ai: Click the "Screenshots" tab or view in the detail panel if not already selected
11+
- sleep: 1000
12+
- aiAssert: The detail panel shows screenshot images of the UI context or execution screenshots
13+
14+
- name: view JSON data tab
15+
flow:
16+
- ai: Click the "JSON View" tab in the detail panel
17+
- sleep: 1000
18+
- aiAssert: The detail panel shows JSON formatted data with curly braces and key-value pairs
19+
20+
- name: expand detail side sections
21+
flow:
22+
- ai: Click on the "Param" section header in the right detail side panel to expand it
23+
- sleep: 1000
24+
- aiAssert: The Param section is expanded showing task parameters or input data
25+
- ai: Click on the "Meta" section header in the right detail side panel to expand it
26+
- sleep: 1000
27+
- aiAssert: The Meta section shows execution metadata like Type, Status, Start time, End time, or Total time
28+
29+
- name: open in playground button
30+
flow:
31+
- ai: Click on a task row in the sidebar that shows "Planning" type
32+
- sleep: 2000
33+
- aiAssert: There is a "Open in Playground" button visible on the page

‎apps/report/e2e/sidebar-navigation.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
target:
2+
serve: ./dist
3+
url: demo.html
4+
tasks:
5+
- name: sidebar task list is rendered
6+
flow:
7+
- sleep: 3000
8+
- aiAssert: The left sidebar shows a list of task execution rows
9+
10+
- name: click a task row to view details
11+
flow:
12+
- ai: Click on the first task row in the left sidebar execution list
13+
- sleep: 2000
14+
- aiAssert: The right side shows a detail panel with task information, not the video player
15+
16+
- name: navigate between tasks
17+
flow:
18+
- ai: Click on the second task row in the left sidebar
19+
- sleep: 1000
20+
- aiAssert: The detail panel content has updated to show task information
21+
22+
- name: toggle model call details
23+
flow:
24+
- ai: Click the "Model Call Details" checkbox or toggle in the sidebar
25+
- sleep: 1000
26+
- aiAssert: The sidebar shows additional columns like Model or Tokens
27+
- ai: Click the "Model Call Details" checkbox again to turn it off
28+
- sleep: 1000

‎apps/report/e2e/theme-toggle.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
target:
2+
serve: ./dist
3+
url: demo.html
4+
tasks:
5+
- name: toggle to dark mode
6+
flow:
7+
- sleep: 2000
8+
- ai: Click the theme toggle button (sun/moon icon) in the top header area
9+
- sleep: 1000
10+
- aiAssert: The page has switched to dark mode with a dark background color
11+
12+
- name: toggle back to light mode
13+
flow:
14+
- ai: Click the theme toggle button (sun/moon icon) again in the top header area
15+
- sleep: 1000
16+
- aiAssert: The page has switched back to light mode with a light/white background color
17+
18+
- name: version info displayed
19+
flow:
20+
- aiAssert: The top header area shows version information with a version number like "v" followed by digits

‎apps/report/e2e/timeline-interaction.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
target:
2+
serve: ./dist
3+
url: demo.html
4+
tasks:
5+
- name: timeline is visible
6+
flow:
7+
- sleep: 2000
8+
- aiAssert: A timeline or "Record" section is visible at the bottom of the page with screenshot thumbnails or a time axis
9+
10+
- name: collapse and expand timeline
11+
flow:
12+
- ai: Click the "Record" header or the collapse/expand arrow on the timeline section
13+
- sleep: 1000
14+
- aiAssert: The timeline section is collapsed and the screenshot thumbnails are hidden
15+
- ai: Click the "Record" header or the expand arrow again to re-expand the timeline
16+
- sleep: 1000
17+
- aiAssert: The timeline section is expanded again showing screenshot thumbnails or time axis
18+
19+
- name: click timeline screenshot to navigate
20+
flow:
21+
- ai: Click on a screenshot thumbnail in the timeline area
22+
- sleep: 2000
23+
- aiAssert: The sidebar highlights a task row and the detail panel shows information for the selected task

0 commit comments

Comments
 (0)