feat(ios): support iOS devices by using WebDriver (#1236)

* feat(ios): implement iOS device and agent management

* feat(ios): switch from simctl to idb for screenshot and gesture automation
fix(tests): update error handling and improve test actions for iOS settings

* feat(ios): enhance text input handling and update test for keyboard behavior

* feat(ios): refactor code structure for improved readability and maintainability

* refactor(ios): remove WebDriverAgent-specific utilities and streamline iOS device management

* feat(ios): enhance iOS automation setup and testing

* feat(ios): improve key press handling for iOS keyboard interactions

* feat(android, ios): add playground scripts and improve device management for Android and iOS

* feat(ios): update playground script to enable debugging and enhance iOS key handling

* feat(docs): add iOS integration sections for WebDriverAgent in English and Chinese documentation

* test(ios): enhance iOS testing suite and improve package structure

* feat(ios): enhance URL launching capabilities and improve documentation for iOS integration

* feat(ios): refactor exec usage to execFile for improved command execution in utils and WDA backend

* feat(ios): update session capabilities to connect to active app and disable idle wait

* feat(ios): update documentation and code to use deviceId instead of udid for iOS integration

* refactor(ios): update iOS package structure tests to use default parameters

* feat(ios): refactor code structure for improved readability and maintainability

* refactor(tests): remove unused import of globalConfigManager in agent.test.ts

* test(ios): refactor code structure for improved readability and maintainability

* feat(ios): implement swipe gesture to dismiss keyboard in IOSWebDriverClient

* feat(ios): enhance iOS automation documentation and configuration examples

* feat(ios): update keyboard dismissal methods to support key names and improve functionality

* test(ios): update keyboard dismissal tests to use swipe gesture and improve error handling

* docs(ios): update integration documentation for WebDriver and Midscene, enhancing clarity and structure

Author: quanru

.gitignore

@@ -113,6 +113,8 @@ midscene_run/dump
 extension_output
 .cursor
 packages/android-playground/static/
+packages/ios-playground/static/
+packages/ios/static/
 packages/playground/static/
 .cursor
 CLAUDE.md

README.md

@@ -44,6 +44,7 @@ English | [简体中文](./README.zh.md)
 ### Web & Mobile App & Any Interface
 - **Web Automation 🖥️**: Either integrate with [Puppeteer](https://midscenejs.com/integrate-with-puppeteer.html), [Playwright](https://midscenejs.com/integrate-with-playwright.html) or use [Bridge Mode](https://midscenejs.com/bridge-mode-by-chrome-extension.html) to control your desktop browser.
 - **Android Automation 📱**: Use [Javascript SDK](https://midscenejs.com/integrate-with-android.html) with adb to control your local Android device.
+- **iOS Automation 🍎**: Use [Javascript SDK](https://midscenejs.com/integrate-with-ios.html) with iOS Simulator to control your local iOS devices and simulators.
 - **Any Interface Automation 🌐**: Use [Javascript SDK](https://midscenejs.com/integrate-with-any-interface.html) to control your own interface.
 
 ### Tools
@@ -135,6 +136,7 @@ We would like to thank the following projects:
 - [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) for the open-source VL model Qwen2.5-VL.
 - [scrcpy](https://github.com/Genymobile/scrcpy) and [yume-chan](https://github.com/yume-chan) allow us to control Android devices with browser.
 - [appium-adb](https://github.com/appium/appium-adb) for the javascript bridge of adb.
+- [appium-webdriveragent](https://github.com/appium/WebDriverAgent) for the javascript operate XCTest。
 - [YADB](https://github.com/ysbing/YADB) for the yadb tool which improves the performance of text input.
 - [Puppeteer](https://github.com/puppeteer/puppeteer) for browser automation and control.
 - [Playwright](https://github.com/microsoft/playwright) for browser automation and control and testing.

README.zh.md

@@ -136,6 +136,7 @@ for (const record of recordList) {
 - [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) 用于开源的视觉语言模型 Qwen2.5-VL。
 - [scrcpy](https://github.com/Genymobile/scrcpy) 和 [yume-chan](https://github.com/yume-chan) 允许我们使用浏览器控制 Android 设备。
 - [appium-adb](https://github.com/appium/appium-adb) 用于 javascript 桥接 adb。
+- [appium-webdriveragent](https://github.com/appium/WebDriverAgent) 用于 javascript 操作 XCTest。
 - [YADB](https://github.com/ysbing/YADB) 用于提高文本输入的兼容性。
 - [Puppeteer](https://github.com/puppeteer/puppeteer) 用于浏览器自动化与控制。
 - [Playwright](https://github.com/microsoft/playwright) 用于浏览器自动化与控制和测试。

apps/android-playground/rsbuild.config.ts

@@ -1,5 +1,5 @@
-import fs from 'node:fs';
 import path from 'node:path';
+import { createPlaygroundCopyPlugin } from '@midscene/shared';
 import { defineConfig } from '@rsbuild/core';
 import { pluginLess } from '@rsbuild/plugin-less';
 import { pluginNodePolyfill } from '@rsbuild/plugin-node-polyfill';
@@ -8,34 +8,6 @@ import { pluginSvgr } from '@rsbuild/plugin-svgr';
 import { pluginTypeCheck } from '@rsbuild/plugin-type-check';
 import { version as playgroundVersion } from '../../packages/playground/package.json';
 
-const copyAndroidPlaygroundStatic = () => ({
-  name: 'copy-android-playground-static',
-  setup(api) {
-    api.onAfterBuild(async () => {
-      const srcDir = path.join(__dirname, 'dist');
-      const destDir = path.join(
-        __dirname,
-        '..',
-        '..',
-        'packages',
-        'android-playground',
-        'static',
-      );
-      const faviconSrc = path.join(__dirname, 'src', 'favicon.ico');
-      const faviconDest = path.join(destDir, 'favicon.ico');
-
-      await fs.promises.mkdir(destDir, { recursive: true });
-      // Copy directory contents recursively
-      await fs.promises.cp(srcDir, destDir, { recursive: true });
-      // Copy favicon
-      await fs.promises.copyFile(faviconSrc, faviconDest);
-
-      console.log(`Copied build artifacts to ${destDir}`);
-      console.log(`Copied favicon to ${faviconDest}`);
-    });
-  },
-});
-
 export default defineConfig({
   environments: {
     web: {
@@ -82,7 +54,12 @@ export default defineConfig({
     pluginNodePolyfill(),
     pluginLess(),
     pluginSvgr(),
-    copyAndroidPlaygroundStatic(),
+    createPlaygroundCopyPlugin(
+      path.join(__dirname, 'dist'),
+      path.join(__dirname, '../../packages/android-playground/static'),
+      'copy-android-playground-static',
+      path.join(__dirname, 'src', 'favicon.ico'),
+    ),
     pluginTypeCheck(),
   ],
 });

apps/playground/rsbuild.config.ts

@@ -1,5 +1,5 @@
-import fs from 'node:fs';
 import path from 'node:path';
+import { createPlaygroundCopyPlugin } from '@midscene/shared';
 import { defineConfig } from '@rsbuild/core';
 import { pluginLess } from '@rsbuild/plugin-less';
 import { pluginNodePolyfill } from '@rsbuild/plugin-node-polyfill';
@@ -8,41 +8,24 @@ import { pluginSvgr } from '@rsbuild/plugin-svgr';
 import { pluginTypeCheck } from '@rsbuild/plugin-type-check';
 import { version as playgroundVersion } from '../../packages/playground/package.json';
 
-const copyWebPlaygroundStatic = () => ({
-  name: 'copy-playground-static',
-  setup(api) {
-    api.onAfterBuild(async () => {
-      const srcDir = path.join(__dirname, 'dist');
-      const destDir = path.join(
-        __dirname,
-        '..',
-        '..',
-        'packages',
-        'playground',
-        'static',
-      );
-      const faviconSrc = path.join(__dirname, 'src', 'favicon.ico');
-      const faviconDest = path.join(destDir, 'favicon.ico');
-
-      await fs.promises.mkdir(destDir, { recursive: true });
-      // Copy directory contents recursively
-      await fs.promises.cp(srcDir, destDir, { recursive: true });
-      // Copy favicon
-      await fs.promises.copyFile(faviconSrc, faviconDest);
-
-      console.log(`Copied build artifacts to ${destDir}`);
-      console.log(`Copied favicon to ${faviconDest}`);
-    });
-  },
-});
-
 export default defineConfig({
   plugins: [
     pluginReact(),
     pluginLess(),
     pluginNodePolyfill(),
     pluginSvgr(),
-    copyWebPlaygroundStatic(),
+    createPlaygroundCopyPlugin(
+      path.join(__dirname, 'dist'),
+      path.join(__dirname, '../../packages/playground/static'),
+      'copy-playground-static',
+      path.join(__dirname, 'src', 'favicon.ico'),
+    ),
+    createPlaygroundCopyPlugin(
+      path.join(__dirname, 'dist'),
+      path.join(__dirname, '../../packages/ios/static'),
+      'copy-ios-playground-static',
+      path.join(__dirname, 'src', 'favicon.ico'),
+    ),
     pluginTypeCheck(),
   ],
   resolve: {

apps/site/docs/en/automate-with-scripts-in-yaml.mdx

@@ -80,6 +80,22 @@ tasks:
       - aiAssert: The results show weather information
 ```
 
+Or, to drive an iOS device automation task (requires WebDriverAgent configuration):
+
+```yaml
+ios:
+  # launch: com.apple.mobilesafari
+  wdaPort: 8100
+
+tasks:
+  - name: Search for weather
+    flow:
+      - ai: Open the browser and navigate to bing.com
+      - ai: Search for "today's weather"
+      - sleep: 3000
+      - aiAssert: The results show weather information
+```
+
 Run the script:
 
 ```bash
@@ -94,7 +110,7 @@ You will see the script's execution progress and the visual report file.
 
 Script files use YAML format to describe automation tasks. It defines the target to be manipulated (like a webpage or an Android app) and the series of steps to perform.
 
-A standard `.yaml` script file includes a `web` or `android` section to configure the environment, and a `tasks` section to define the automation tasks.
+A standard `.yaml` script file includes a `web`, `android`, or `ios` section to configure the environment, and a `tasks` section to define the automation tasks.
 
 ```yaml
 web:
@@ -177,6 +193,29 @@ android:
   output: <path-to-output-file>
 ```
 
+### The `ios` part
+
+```yaml
+ios:
+  # WebDriverAgent port, optional, defaults to 8100.
+  wdaPort: <port>
+
+  # WebDriverAgent host address, optional, defaults to localhost.
+  wdaHost: <host>
+
+  # Whether to auto dismiss keyboard, optional, defaults to false.
+  autoDismissKeyboard: <boolean>
+
+  # Launch URL or app bundle ID, optional, defaults to the device's current page.
+  launch: <url-or-bundle-id>
+
+  # The path to the JSON file for outputting aiQuery/aiAssert results, optional.
+  output: <path-to-output-file>
+
+  # Whether to save log content to a JSON file, optional, defaults to `false`. If true, saves to `unstableLogContent.json`. If a string, saves to the specified path. The log content structure may change in the future.
+  unstableLogContent: <boolean | path-to-unstable-log-file>
+```
+
 ### The `tasks` part
 
 The `tasks` part is an array that defines the steps of the script. Remember to add a `-` before each step to indicate it's an array item.
@@ -352,6 +391,8 @@ The command-line tool provides several options to control the execution behavior
 - `--web.viewportWidth <width>`: Sets the browser viewport width, which will override the `web.viewportWidth` parameter in all script files.
 - `--web.viewportHeight <height>`: Sets the browser viewport height, which will override the `web.viewportHeight` parameter in all script files.
 - `--android.deviceId <device-id>`: Sets the Android device ID, which will override the `android.deviceId` parameter in all script files.
+- `--ios.wdaPort <port>`: Sets the WebDriverAgent port, which will override the `ios.wdaPort` parameter in all script files.
+- `--ios.wdaHost <host>`: Sets the WebDriverAgent host address, which will override the `ios.wdaHost` parameter in all script files.
 - `--dotenv-debug`: Sets the debug log for dotenv, disabled by default.
 - `--dotenv-override`: Sets whether dotenv overrides global environment variables with the same name, disabled by default.
 

apps/site/docs/en/integrate-with-android.mdx

@@ -17,11 +17,13 @@ Integrate Vitest for testing: [https://github.com/web-infra-dev/midscene-example
 
 <SetupEnv />
 
-## Step 1. Install dependencies
+## Integrate Midscene
+
+### Step 1: Install dependencies
 
 <PackageManagerTabs command="install @midscene/android --save-dev" />
 
-## Step 2. Write scripts
+### Step 2: Write scripts
 
 Let's take a simple example: search for headphones on eBay using the browser in the Android device. （Of course, you can also use any other apps on the Android device.）
 
@@ -72,7 +74,7 @@ Promise.resolve(
 );
 ```
 
-## Step 3. Run
+### Step 3: Run
 
 Using `tsx` to run
 
@@ -96,11 +98,13 @@ After a while, you will see the following output:
 ]
 ```
 
-## Step 4: View the report
+### Step 4: View the report
 
 After the above command executes successfully, the console will output: `Midscene - report file updated: /path/to/report/some_id.html`. You can open this file in a browser to view the report.
 
-## `AndroidDevice` constructor
+## Constructor and Interface
+
+### `AndroidDevice` Constructor
 
 The AndroidDevice constructor supports the following parameters:
 
@@ -114,11 +118,11 @@ The AndroidDevice constructor supports the following parameters:
   - `imeStrategy?: 'always-yadb' | 'yadb-for-non-ascii'` - Optional, when should Midscene invoke [yadb](https://github.com/ysbing/YADB) to input texts. (Default: 'always-yadb')
   - `displayId?: number` - Optional, the display id to use. (Default: undefined, means use the current display)
 
-## More interfaces in AndroidAgent
+### Additional Android Agent Interfaces
 
 Except the common agent interfaces in [API Reference](./api.mdx), AndroidAgent also provides some other interfaces:
 
-### `agent.launch()`
+#### `agent.launch()`
 
 Launch a webpage or native page.
 
@@ -149,7 +153,7 @@ await agent.launch('com.android.settings'); // open a native page
 await agent.launch('com.android.settings/.Settings'); // open a native page
 ```
 
-### `agentFromAdbDevice()`
+#### `agentFromAdbDevice()`
 
 Create a AndroidAgent from a connected adb device.
 
@@ -180,7 +184,7 @@ const agent = await agentFromAdbDevice('s4ey59'); // create a AndroidAgent from
 const agent = await agentFromAdbDevice(); // no deviceId, use the first connected device
 ```
 
-### `getConnectedDevices()`
+#### `getConnectedDevices()`
 
 Get all connected Android devices.
 
@@ -216,7 +220,7 @@ console.log(devices);
 const agent = await agentFromAdbDevice(devices[0].udid);
 ```
 
-## Provide custom actions
+## Extending Custom Interaction Actions
 
 Use the `customActions` option to extend the agent's action space with your own actions defined via `defineAction`. When provided, these actions will be appended to the built-in ones so the agent can call them during planning.
 

apps/site/docs/en/integrate-with-any-interface.mdx

@@ -30,6 +30,8 @@ We have prepared a demo project for you to learn how to define your own interfac
 
 * [Android (adb) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/android/src/device.ts) - This is the Android (adb) Agent for Midscene that implements this feature
 
+* [iOS (WebDriverAgent) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/ios/src/device.ts) - This is the iOS (WebDriverAgent) Agent for Midscene that implements this feature
+
 There are also some community projects that use this feature:
 
 * [midscene-ios](https://github.com/lhuanyu/midscene-ios) - A project driving the OSX "iPhone Mirroring" app with Midscene