Skip to content

Commit 4190529

Browse files
quanruquanru
authored
feat(computer): add Electron demo for testing Obsidian on headless Linux CI (#87)
* feat(computer): add Electron demo for testing Obsidian on headless Linux CI Add a new computer/electron-demo that uses @midscene/computer to test Obsidian (Electron app) in a Xvfb headless Linux environment. Includes a workflow that downloads the AppImage, extracts it, and runs the demo. * fix(computer): connect agent before launching Obsidian to ensure Xvfb is ready agentFromComputer() starts Xvfb and sets DISPLAY on headless Linux. Obsidian must be launched after this, otherwise it has no display. * ci(computer): add push trigger for electron-demo workflow Enables automatic CI runs when electron-demo files or the workflow itself are pushed to main. * ci(computer): trigger workflow on feature branch push for testing * fix(computer): force process exit after demo completes The Node.js process hung after demo succeeded because Obsidian child processes kept it alive. Use SIGKILL + process.exit(0) to ensure clean exit. * fix(computer): handle XIO fatal error on process exit - Use SIGKILL to self-terminate without triggering Xlib cleanup - Check demo output log for success string instead of relying on exit code * feat(computer): add LifeOS community plugin installation to Electron demo
1 parent 01b4f5f commit 4190529

File tree

5 files changed

+355
-0
lines changed

5 files changed

+355
-0
lines changed

‎.github/workflows/computer-electron-demo.yaml‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
name: Computer Electron Demo
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
workflow_dispatch:
8+
inputs:
9+
tag:
10+
description: 'The tag name of @midscene/computer'
11+
required: true
12+
default: 'beta'
13+
branch:
14+
description: 'The branch to checkout'
15+
required: false
16+
default: 'main'
17+
push:
18+
branches:
19+
- main
20+
- feat/computer-electron-demo
21+
paths:
22+
- 'computer/electron-demo/**'
23+
- '.github/workflows/computer-electron-demo.yaml'
24+
25+
env:
26+
TAG: ${{ github.event.inputs.tag || 'beta' }}
27+
28+
jobs:
29+
electron-demo:
30+
runs-on: ubuntu-22.04
31+
env:
32+
OPENAI_BASE_URL: ${{ secrets.QWEN_OPENAI_BASE_URL }}
33+
OPENAI_API_KEY: ${{ secrets.QWEN_OPENAI_API_KEY }}
34+
MIDSCENE_MODEL_NAME: 'qwen-vl-max-latest'
35+
MIDSCENE_USE_QWEN_VL: 1
36+
MIDSCENE_COMPUTER_HEADLESS_LINUX: 'true'
37+
DEBUG: 'midscene:ai:profile:*'
38+
39+
steps:
40+
- name: Checkout repository
41+
uses: actions/checkout@v4
42+
with:
43+
ref: ${{ github.event.inputs.branch || github.ref }}
44+
45+
- name: Install system dependencies
46+
run: |
47+
sudo apt-get update
48+
sudo apt-get install -y \
49+
xvfb \
50+
x11-xserver-utils \
51+
imagemagick \
52+
libxtst6 \
53+
libxinerama1 \
54+
libx11-6 \
55+
libxkbcommon-x11-0 \
56+
libpng16-16 \
57+
libnss3 \
58+
libatk-bridge2.0-0 \
59+
libdrm2 \
60+
libgbm1 \
61+
libasound2 \
62+
libgtk-3-0 \
63+
libnotify4 \
64+
libsecret-1-0 \
65+
libxss1 \
66+
xdg-utils
67+
68+
- name: Download Obsidian AppImage
69+
working-directory: computer/electron-demo
70+
run: |
71+
OBSIDIAN_VERSION="1.8.9"
72+
OBSIDIAN_URL="https://github.com/obsidianmd/obsidian-releases/releases/download/v${OBSIDIAN_VERSION}/Obsidian-${OBSIDIAN_VERSION}.AppImage"
73+
echo "Downloading Obsidian ${OBSIDIAN_VERSION}..."
74+
wget -q "${OBSIDIAN_URL}" -O "Obsidian-${OBSIDIAN_VERSION}.AppImage"
75+
chmod +x "Obsidian-${OBSIDIAN_VERSION}.AppImage"
76+
77+
- name: Install dependencies and run demo
78+
id: run-demo
79+
working-directory: computer/electron-demo
80+
run: |
81+
npm i pnpm -g
82+
pnpm i @midscene/computer@${{ env.TAG }} --save-dev
83+
pnpm install
84+
pnpm run test 2>&1 | tee /tmp/demo-output.log || true
85+
86+
- name: Upload report
87+
if: always()
88+
uses: actions/upload-artifact@v4
89+
with:
90+
if-no-files-found: ignore
91+
name: Electron Demo Report
92+
path: computer/electron-demo/midscene_run/report
93+
94+
- name: Check if demo succeeded
95+
run: grep -q "Electron demo completed successfully" /tmp/demo-output.log

‎computer/electron-demo/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
node_modules
2+
.env
3+
midscene_run
4+
obsidian-vault/
5+
*.AppImage
6+
squashfs-root/

‎computer/electron-demo/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Electron Demo — Testing Obsidian with @midscene/computer
2+
3+
This demo shows how to use `@midscene/computer` to test an Electron application (Obsidian) in a headless Linux CI environment using Xvfb.
4+
5+
## How it works
6+
7+
1. Finds or extracts the Obsidian AppImage (`--appimage-extract` to avoid FUSE)
8+
2. Pre-seeds the vault config to skip the vault picker dialog
9+
3. Launches Obsidian with Electron-friendly flags (`--no-sandbox`, etc.)
10+
4. Uses `agentFromComputer()` to interact with the running app via screenshots
11+
12+
## Local usage
13+
14+
```bash
15+
# 1. Download Obsidian AppImage into this directory
16+
# 2. Create .env with your AI model credentials:
17+
# OPENAI_API_KEY=sk-xxx
18+
# MIDSCENE_MODEL_NAME=qwen3-vl-plus
19+
# 3. Run
20+
npm install
21+
npm run test
22+
```
23+
24+
## CI usage
25+
26+
The workflow `.github/workflows/computer-electron-demo.yaml` runs this demo on `ubuntu-22.04` with Xvfb. Trigger it manually via `workflow_dispatch`.

‎computer/electron-demo/demo.ts‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
import { execSync, spawn } from 'node:child_process';
2+
import { existsSync, mkdirSync, readdirSync, writeFileSync } from 'node:fs';
3+
import { homedir } from 'node:os';
4+
import { join, resolve } from 'node:path';
5+
import { agentFromComputer } from '@midscene/computer';
6+
import 'dotenv/config';
7+
8+
const sleep = (ms: number) =>
9+
new Promise((resolve) => setTimeout(resolve, ms));
10+
11+
const DEMO_DIR = resolve(import.meta.dirname);
12+
const VAULT_DIR = join(DEMO_DIR, 'obsidian-vault');
13+
14+
/**
15+
* Locate the Obsidian binary. If only an AppImage is found, extract it first.
16+
* Returns the path to the executable.
17+
*/
18+
function findOrPrepareApp(): string {
19+
// Already extracted
20+
const extractedBin = join(DEMO_DIR, 'squashfs-root', 'obsidian');
21+
if (existsSync(extractedBin)) {
22+
console.log('Using previously extracted Obsidian binary');
23+
return extractedBin;
24+
}
25+
26+
// Find AppImage in the demo directory
27+
const appImages = readdirSync(DEMO_DIR).filter((f) =>
28+
f.endsWith('.AppImage'),
29+
);
30+
if (appImages.length === 0) {
31+
throw new Error(
32+
'No Obsidian AppImage found. Download it to the electron-demo directory first.',
33+
);
34+
}
35+
36+
const appImagePath = join(DEMO_DIR, appImages[0]);
37+
console.log(`Extracting ${appImages[0]} ...`);
38+
execSync(`chmod +x "${appImagePath}"`);
39+
execSync(`"${appImagePath}" --appimage-extract`, { cwd: DEMO_DIR });
40+
console.log('Extraction complete');
41+
42+
if (!existsSync(extractedBin)) {
43+
throw new Error('Extraction succeeded but obsidian binary not found');
44+
}
45+
return extractedBin;
46+
}
47+
48+
/**
49+
* Pre-seed Obsidian config so it opens our vault directly (skipping the vault picker).
50+
*/
51+
function preseedVault(vaultDir: string): void {
52+
mkdirSync(vaultDir, { recursive: true });
53+
54+
const configDir = join(homedir(), '.config', 'obsidian');
55+
mkdirSync(configDir, { recursive: true });
56+
57+
const obsidianJson = join(configDir, 'obsidian.json');
58+
const vaultId = 'ci-test-vault';
59+
const config = {
60+
vaults: {
61+
[vaultId]: {
62+
path: vaultDir,
63+
ts: Date.now(),
64+
open: true,
65+
},
66+
},
67+
};
68+
writeFileSync(obsidianJson, JSON.stringify(config, null, 2));
69+
console.log(`Vault config written to ${obsidianJson}`);
70+
}
71+
72+
/**
73+
* Launch Obsidian as a detached child process with Electron-friendly flags.
74+
*/
75+
function launchApp(
76+
binaryPath: string,
77+
vaultDir: string,
78+
): ReturnType<typeof spawn> {
79+
const args = [
80+
`--vault=${vaultDir}`,
81+
'--no-sandbox',
82+
'--disable-gpu',
83+
'--disable-dev-shm-usage',
84+
];
85+
console.log(`Launching: ${binaryPath} ${args.join(' ')}`);
86+
87+
const child = spawn(binaryPath, args, {
88+
detached: true,
89+
stdio: 'ignore',
90+
env: { ...process.env, ELECTRON_DISABLE_SECURITY_WARNINGS: 'true' },
91+
});
92+
child.unref();
93+
return child;
94+
}
95+
96+
(async () => {
97+
// --- Prepare app binary & vault config ---
98+
const binaryPath = findOrPrepareApp();
99+
preseedVault(VAULT_DIR);
100+
101+
// --- Connect Midscene agent FIRST (this starts Xvfb on headless Linux) ---
102+
const agent = await agentFromComputer({
103+
aiActionContext:
104+
'You are interacting with Obsidian, a note-taking desktop application. ' +
105+
'If any dialog or popup appears, dismiss it by clicking the close button or pressing Escape.',
106+
});
107+
108+
// --- Launch Obsidian AFTER Xvfb is ready (DISPLAY is now set) ---
109+
const child = launchApp(binaryPath, VAULT_DIR);
110+
console.log(`Obsidian launched (pid: ${child.pid})`);
111+
112+
// Give Obsidian time to start up
113+
await sleep(10000);
114+
115+
try {
116+
// Wait for the main UI to appear
117+
await agent.aiWaitFor(
118+
'Obsidian main window is visible with the editor area or vault view',
119+
{ timeoutMs: 30000 },
120+
);
121+
console.log('Obsidian UI is ready');
122+
123+
// Dismiss any welcome dialogs / popups
124+
await agent.aiAct(
125+
'If there is any popup, modal dialog, or "Trust author" prompt, close or dismiss it',
126+
);
127+
await sleep(1000);
128+
129+
// Create a new note
130+
await agent.aiAct('press Ctrl+N to create a new note');
131+
await sleep(1500);
132+
133+
// Type note content
134+
await agent.aiAct('type "Hello from Midscene CI" in the editor');
135+
await sleep(1000);
136+
137+
// Verify the note content
138+
const result = await agent.aiQuery(
139+
'{ content: string } — read the text content currently visible in the editor area',
140+
);
141+
console.log('Editor content:', JSON.stringify(result));
142+
143+
if (
144+
typeof result?.content === 'string' &&
145+
result.content.includes('Midscene')
146+
) {
147+
console.log('Content verification passed!');
148+
} else {
149+
console.warn('Content verification: text may not match expected value');
150+
}
151+
152+
// --- Install community plugin: LifeOS ---
153+
console.log('Opening Obsidian settings...');
154+
await agent.aiAct('press Ctrl+Comma to open Settings');
155+
await sleep(2000);
156+
157+
await agent.aiWaitFor('Settings dialog is visible', { timeoutMs: 15000 });
158+
159+
// Navigate to Community plugins
160+
await agent.aiAct(
161+
'click "Community plugins" in the left sidebar of the settings dialog',
162+
);
163+
await sleep(1500);
164+
165+
// Turn on community plugins if not enabled
166+
await agent.aiAct(
167+
'If there is a "Turn on community plugins" button, click it. ' +
168+
'If a confirmation dialog appears, click "Turn on" to confirm.',
169+
);
170+
await sleep(1500);
171+
172+
// Open the plugin browser
173+
await agent.aiAct('click "Browse" button to open the community plugin browser');
174+
await sleep(3000);
175+
176+
await agent.aiWaitFor(
177+
'Community plugin browser / marketplace is visible with a search box',
178+
{ timeoutMs: 20000 },
179+
);
180+
console.log('Community plugin browser is open');
181+
182+
// Search for LifeOS
183+
await agent.aiAct('type "lifeos" in the search box');
184+
await sleep(3000);
185+
186+
// Click the LifeOS plugin from results
187+
await agent.aiAct('click on the "LifeOS" plugin in the search results');
188+
await sleep(2000);
189+
190+
// Install the plugin
191+
await agent.aiAct('click the "Install" button');
192+
await sleep(5000);
193+
194+
// Verify installation
195+
await agent.aiWaitFor(
196+
'The "Install" button has changed to "Enable" or "Installed", indicating the plugin was installed',
197+
{ timeoutMs: 30000 },
198+
);
199+
console.log('LifeOS plugin installed successfully!');
200+
201+
console.log('Electron demo completed successfully!');
202+
} finally {
203+
if (child.pid) {
204+
try {
205+
process.kill(-child.pid, 'SIGKILL');
206+
} catch {
207+
// Process may have already exited
208+
}
209+
}
210+
// Force immediate exit without triggering Xlib cleanup (which causes XIO fatal error)
211+
setTimeout(() => process.kill(process.pid, 'SIGKILL'), 500);
212+
}
213+
})();

‎computer/electron-demo/package.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"name": "computer-electron-demo",
3+
"private": true,
4+
"version": "1.0.0",
5+
"description": "Demo for testing Electron apps (e.g. Obsidian) with @midscene/computer",
6+
"type": "module",
7+
"scripts": {
8+
"test": "tsx demo.ts"
9+
},
10+
"devDependencies": {
11+
"@midscene/computer": "latest",
12+
"dotenv": "^16.4.5",
13+
"tsx": "4.20.1"
14+
}
15+
}

0 commit comments

Comments
 (0)