Add accessbility debugging rule

Author: steipete

.cursor/rules/ax.mdc

@@ -0,0 +1,200 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# macOS Accessibility (`ax`) Binary Rules & Knowledge
+
+This document outlines the functionality, build process, testing procedures, and technical details of the `ax` Swift command-line utility, designed for interacting with the macOS Accessibility framework.
+
+## 1. `ax` Binary Overview
+
+*   **Purpose**: Provides a JSON-based interface to query UI elements and perform actions using the macOS Accessibility API. It's intended to be called by other processes (like the MCP server).
+*   **Communication**: Operates by reading JSON commands from `stdin` and writing JSON responses (or errors) to `stdout` (or `stderr` for errors).
+*   **Core Commands**:
+    *   `query`: Retrieves information about UI elements.
+    *   `perform`: Executes an action on a UI element.
+*   **Key Input Fields (JSON)**:
+    *   `cmd` (string): "query" or "perform".
+    *   `locator` (object): Specifies the target element(s).
+        *   `app` (string): Bundle ID or localized name of the target application (e.g., "com.apple.TextEdit", "Safari").
+        *   `role` (string): The accessibility role of the target element (e.g., "AXWindow", "AXButton", "*").
+        *   `match` (object): Key-value pairs of attributes to match (e.g., `{"AXMain": "true"}`). Values are strings.
+        *   `pathHint` (array of strings, optional): A path to navigate the UI tree (e.g., `["window[1]", "toolbar[1]"]`).
+    *   `attributes` (array of strings, optional): For `query`, specific attributes to retrieve. Defaults to a common set if omitted.
+    *   `action` (string, optional): For `perform`, the action to execute (e.g., "AXPress").
+    *   `multi` (boolean, optional): For `query`, if `true`, returns all matching elements. Defaults to `false`.
+    *   `requireAction` (string, optional): For `query`, filters results to elements supporting a specific action.
+    *   `debug_logging` (boolean, optional): If `true`, includes detailed internal debug logs in the response.
+*   **Key Output Fields (JSON)**:
+    *   Success (`query`): `{ "attributes": { "AXTitle": "...", ... } }`
+    *   Success (`query`, `multi: true`): `{ "elements": [ { "AXTitle": "..." }, ... ] }`
+    *   Success (`perform`): `{ "status": "ok" }`
+    *   Error: `{ "error": "Error message description" }`
+    *   `debug_logs` (array of strings, optional): Included in success or error responses if `debug_logging` was true.
+
+## 2. Functionality - How it Works
+
+The `ax` binary is implemented in Swift in `ax/Sources/AXHelper/main.swift`.
+
+*   **Application Targeting**:
+    *   `getApplicationElement(bundleIdOrName: String)`: This function is the entry point to an application's accessibility tree.
+        *   It first tries to find the application using its bundle identifier (e.g., "com.apple.Safari") via `NSRunningApplication.runningApplications(withBundleIdentifier:)`.
+        *   If not found, it iterates through all running applications and attempts to match by the application's localized name (e.g., "Safari") via `NSRunningApplication.localizedName`.
+        *   Once the `NSRunningApplication` instance is found, `AXUIElementCreateApplication(pid)` is used to get the root `AXUIElement` for that application.
+
+*   **Element Location**:
+    *   **`search(element:locator:depth:maxDepth:)`**:
+        *   Used for single-element queries (when `multi` is `false` or not set).
+        *   Performs a depth-first search starting from a given `element` (usually the application element or one found via `pathHint`).
+        *   It checks if an element's `AXRole` matches `locator.role`.
+        *   Then, it verifies that all attribute-value pairs in `locator.match` correspond to the element's actual attributes. This matching logic handles:
+            *   **Boolean attributes** (e.g., `AXMain`, `AXFocused`): Compares against string "true" or "false".
+            *   **Numeric attributes**: Attempts to parse `wantStr` (from `locator.match`) as an `Int` and compares numerically.
+            *   **String attributes**: Performs direct string comparison.
+        *   If a match is found, the `AXUIElement` is returned. Otherwise, it recursively searches children.
+    *   **`collectAll(element:locator:requireAction:hits:depth:maxDepth:)`**:
+        *   Used for multi-element queries (`multi: true`).
+        *   Recursively traverses the accessibility tree starting from `element`.
+        *   Matches elements against `locator.role` (supports `"*"` or empty for wildcard) and `locator.match` (using robust boolean, numeric, and string comparison similar to `search`).
+        *   If `requireAction` is specified, it further filters elements to those supporting the given action using `elementSupportsAction`.
+        *   It aggregates all matching `AXUIElement`s into the `hits` array.
+        *   To discover children, it queries a comprehensive list of attributes known to contain child elements:
+            *   Standard: `kAXChildrenAttribute` ("AXChildren")
+            *   Web-specific: "AXLinks", "AXButtons", "AXControls", "AXDOMChildren", etc.
+            *   Application-specific: `kAXWindowsAttribute` ("AXWindows")
+            *   General containers: "AXContents", "AXVisibleChildren", etc.
+        *   Includes deduplication of found elements based on their `ObjectIdentifier`.
+    *   **`navigateToElement(from:pathHint:)`**:
+        *   Processes the `pathHint` array (e.g., `["window[1]", "toolbar[1]"]`).
+        *   Each component (e.g., "window[1]") is parsed into a role ("window") and a 0-based index (0).
+        *   It navigates the tree by finding children of the current element that match the role and selecting the one at the specified index.
+        *   Special handling for "window" role uses the `AXWindows` attribute for direct access.
+        *   The element found at the end of the path is used as the starting point for `search` or `collectAll`.
+
+*   **Attribute Retrieval**:
+    *   `getElementAttributes(element:attributes:)`: Fetches attributes for a given `AXUIElement`.
+        *   If the input `attributes` list is empty or nil, it discovers all available attributes for the element using `AXUIElementCopyAttributeNames`.
+        *   It then iterates through the attributes to retrieve their values using `AXUIElementCopyAttributeValue`.
+        *   Handles various `CFTypeRef` return types and converts them to Swift/JSON-compatible representations:
+            *   `CFString` -> `String`
+            *   `CFBoolean` -> `Bool`
+            *   `CFNumber` -> `Int` (or "Number (conversion failed)")
+            *   `CFArray` -> Array of strings (for "AXActions") or descriptive string like "Array with X elements".
+            *   `AXValue` (for `AXPosition`, `AXSize`): Extracts `CGPoint` or `CGSize` and converts to `{"x": Int, "y": Int}` or `{"width": Int, "height": Int}`. Uses `AXValueGetTypeID()`, `AXValueGetType()`, and `AXValueGetValue()`.
+            *   `AXUIElement` (for attributes like `AXTitleUIElement`): Attempts to extract a display string (e.g., its "AXValue" or "AXTitle").
+        *   Includes a `ComputedName` by trying `AXTitle`, `AXTitleUIElement`, `AXValue`, `AXDescription`, `AXLabel`, `AXHelp`, `AXRoleDescription` in order of preference.
+        *   Includes `IsClickable` (boolean) if the element is an `AXButton` or has an `AXPress` action.
+
+*   **Action Performing**:
+    *   `handlePerform(cmd:)` calls `AXUIElementPerformAction(element, actionName)` to execute the specified action on the located element.
+    *   `elementSupportsAction(element:action:)` checks if an element supports a given action by fetching `AXActionNames` and checking for the action's presence.
+
+*   **Error Handling**:
+    *   Uses a custom `AXErrorString` Swift enum (`.notAuthorised`, `.elementNotFound`, `.actionFailed`).
+    *   Responds with a JSON `ErrorResponse` object: `{ "error": "message", "debug_logs": [...] }`.
+
+*   **Debugging**:
+    *   `GLOBAL_DEBUG_ENABLED` (Swift constant, currently `true`): If true, all `debug()` messages are printed to `stderr` of the `ax` process.
+    *   `debug_logging` field in input JSON: If `true`, enables `commandSpecificDebugLoggingEnabled`.
+    *   `collectedDebugLogs` (Swift array): Stores debug messages if `commandSpecificDebugLoggingEnabled` is true. This array is then included in the `debug_logs` field of the JSON response (both success and error).
+    *   The `debug(_ message: String)` function handles appending to `collectedDebugLogs` and printing to `stderr`.
+
+## 3. Build Process & Optimization
+
+The `ax` binary is built using the `Makefile` located in the `ax/` directory.
+
+*   **Makefile (`ax/Makefile`)**:
+    *   **Universal Binary**: Builds for both `arm64` and `x86_64` architectures.
+    *   **Optimization Flags**:
+        *   `-Xswiftc -Osize`: Instructs the Swift compiler to optimize for binary size.
+        *   `-Xlinker -Wl,-dead_strip`: Instructs the linker to perform dead code elimination.
+    *   **Symbol Stripping**:
+        *   `strip -x $(UNIVERSAL_BINARY_PATH)`: Aggressively removes symbols from the linked universal binary to further reduce size.
+    *   **Output**: The final, optimized, and stripped binary is placed at `ax/ax`.
+    *   **Targets**:
+        *   `all` (default): Ensures the old `ax/ax` binary is removed, then builds the new one. It calls `$(MAKE) $(FINAL_BINARY_PATH)` to trigger the dependent build steps.
+        *   `$(FINAL_BINARY_PATH)`: Copies the built and stripped universal binary from the Swift build directory to `ax/ax`.
+        *   `$(UNIVERSAL_BINARY_PATH)`: Contains the `swift build` and `strip` commands.
+        *   `clean`: Removes Swift build artifacts (`.build/`) and the `ax/ax` binary.
+*   **Optimization Journey Summary**:
+    *   The combination of `-Xswiftc -Osize`, `-Xlinker -Wl,-dead_strip`, and `strip -x` proved most effective for size reduction (e.g., from an initial ~369KB down to ~336KB).
+    *   Link-Time Optimization (`-Xswiftc -lto=llvm-full` or `-Xswiftc -lto=llvm-thin`) was attempted but resulted in linker errors (`ld: file cannot be open()ed... main.o`).
+    *   UPX compression was explored. While it significantly reduced size (e.g., 338K to 130K with `--force-macos`), the resulting binary was malformed (`zsh: malformed Mach-o file`) and unusable. UPX was therefore abandoned.
+    *   Other flags like `-Xswiftc -Oz` (not recognized by `swift build`) and `-Xlinker -compress_text` (caused linker errors) were unsuccessful.
+
+## 4. Running & Testing
+
+The `ax` binary is designed to be invoked by a parent process (like the MCP server) but can also be tested manually from the command line.
+
+*   **Runner Script (`ax/ax_runner.sh`)**:
+    *   This is the **recommended way to execute `ax` manually** for testing and debugging.
+    *   It's a simple Bash script that robustly determines its own directory and then executes the `ax/ax` binary, passing along any arguments.
+    *   The TypeScript `AXQueryExecutor.ts` uses this runner script.
+    *   Script content:
+        ```bash
+        #!/bin/bash
+        SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" &> /dev/null && pwd)"
+        exec "$SCRIPT_DIR/ax" "$@"
+        ```
+
+*   **Manual Testing Workflow**:
+    1.  **Ensure Target Application State**: Before running a test, **critically verify** that the target application is running and is in the specific state you intend to query. For example, if you are querying for a window with `AXMain=true`, ensure the application has an actual document window open and focused, not just a file dialog or a menu bar. Mismatched application state is a common reason for "element not found" errors.
+    2.  **Construct JSON Input**: Prepare your command as a single line of JSON.
+    3.  **Execute via `ax_runner.sh`**: Pipe the JSON to the runner script.
+        *   Example:
+            ```bash
+            echo '{"cmd":"query","locator":{"app":"TextEdit","role":"AXWindow","match":{"AXMain":"true"}},"debug_logging":true}' | ./ax/ax_runner.sh
+            ```
+            (You can also run `./ax/ax` directly, but the runner is slightly more robust for scripting.)
+    4.  **Interpret Output**:
+        *   **`stdout`**: Receives the primary JSON response from `ax`. This will be a `QueryResponse`, `MultiQueryResponse`, or `PerformResponse` on success.
+        *   **`stderr`**:
+            *   If `ax` encounters an internal error or fails to parse the input, it will output an `ErrorResponse` JSON to `stderr` (e.g., `{"error":"No element matches the locator","debug_logs":[...]}`).
+            *   If `GLOBAL_DEBUG_ENABLED` is `true` in `main.swift` (which it is by default), all `debug(...)` messages from `ax` are continuously printed to `stderr`, prefixed with `DEBUG:`. This provides a live trace of `ax`'s internal operations.
+            *   The `debug_logs` array within the JSON response (on `stdout` for success, or `stderr` for `ErrorResponse`) contains logs collected specifically for that command if `"debug_logging": true` was in the input JSON.
+
+*   **Example Test Queries**:
+    1.  **Find TextEdit's main window (single element query)**:
+        *Ensure TextEdit is running and has an actual document window open and active.*
+        ```bash
+        echo '{"cmd":"query","locator":{"app":"com.apple.TextEdit","role":"AXWindow","match":{"AXMain":"true"}},"return_all_matches":false,"debug_logging":true}' | ./ax/ax_runner.sh
+        ```
+    2.  **List all elements in TextEdit (multi-element query)**:
+        *Ensure TextEdit is running.*
+        ```bash
+        echo '{"cmd":"query","locator":{"app":"com.apple.TextEdit","role":"*","match":{}},"return_all_matches":true,"debug_logging":true}' | ./ax/ax_runner.sh
+        ```
+
+*   **Permissions**:
+    *   **Crucial**: The application that executes `ax` (e.g., Terminal, your IDE, the Node.js process running the MCP server) **must** have "Accessibility" permissions granted in macOS "System Settings > Privacy & Security > Accessibility".
+    *   The `ax` binary itself calls `checkAccessibilityPermissions()` at startup. If permissions are not granted, it prints detailed instructions to `stderr` and exits.
+
+## 5. macOS Accessibility (AX) Intricacies & Swift Integration
+
+Working with the macOS Accessibility framework via Swift involves several specific considerations:
+
+*   **Frameworks**:
+    *   `ApplicationServices`: Essential for `AXUIElement` and related C APIs.
+    *   `AppKit`: Used for `NSRunningApplication` (to get PIDs) and `NSWorkspace`.
+*   **Element Hierarchy**: UI elements form a tree. Traversal typically involves getting an element's children via attributes like `kAXChildrenAttribute` ("AXChildren"), `kAXWindowsAttribute` ("AXWindows"), etc.
+*   **Attributes (`AX...`)**:
+    *   Elements possess a wide range of attributes (e.g., `AXRole`, `AXTitle`, `AXSubrole`, `AXValue`, `AXFocused`, `AXMain`, `AXPosition`, `AXSize`, `AXIdentifier`). The presence of attributes can vary.
+    *   `CFTypeRef`: Attribute values are returned as `CFTypeRef`. Runtime type checking using `CFGetTypeID()` and `AXValueGetTypeID()` (for `AXValue` types) is necessary before safe casting.
+    *   `AXValue`: A special CoreFoundation type used for geometry (like `CGPoint` for `AXPosition`, `CGSize` for `AXSize`) and other structured data. Requires `AXValueGetValue()` to extract the underlying data.
+*   **Actions (`AX...Action`)**:
+    *   Elements expose supported actions (e.g., `kAXPressAction` ("AXPress"), "AXShowMenu") via the `kAXActionsAttribute` ("AXActions") or `AXUIElementCopyActionNames()`.
+    *   Actions are performed using `AXUIElementPerformAction()`.
+*   **Roles**:
+    *   `AXRole` (e.g., "AXWindow", "AXButton", "AXTextField") and `AXRoleDescription` (a human-readable string) describe the type/function of an element.
+    *   `AXRoleDescription` can sometimes be missing or less reliable than `AXRole`.
+    *   Using `"*"` or an empty string for `locator.role` acts as a wildcard in `collectAll`.
+*   **Data Type Matching**:
+    *   When matching attributes from JSON input (where values are strings), the Swift code must correctly interpret these strings against the actual attribute types (e.g., string "true" for a `Bool` attribute, string "123" for a numeric attribute). Both `search` and `collectAll` implement logic for this.
+*   **Bridging & Constants**:
+    *   Some C-based Accessibility constants (like `kAXWindowsAttribute`) might need to be defined as Swift constants if not directly available.
+    *   Private C functions like `AXUIElementGetTypeID_Impl()` might require `@_silgen_name` bridging.
+*   **Debugging Tool**:
+    *   **Accessibility Inspector** (available in Xcode under "Xcode > Open Developer Tool > Accessibility Inspector") is an indispensable tool for visually exploring the accessibility hierarchy of any running application, viewing element attributes, and testing actions.
+
+This document should serve as a good reference for understanding and working with the `ax` binary.