Skip to content

Commit 2e8b3f7

Browse files
committed
Explain new accessibility tool
1 parent b11d420 commit 2e8b3f7

File tree

1 file changed

+83
-0
lines changed

1 file changed

+83
-0
lines changed

README.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -208,6 +208,89 @@ Retrieves AppleScript/JXA tips, examples, and runnable script details from the s
208208
- Search for tips related to "clipboard":
209209
`{ "toolName": "get_scripting_tips", "input": { "search_term": "clipboard" } }`
210210
211+
### 3. `accessibility_query`
212+
213+
Query and interact with the macOS accessibility interface to inspect UI elements of applications. This tool provides a powerful way to explore and manipulate the user interface elements of any application using the native macOS accessibility framework.
214+
215+
This tool exposes the complete macOS accessibility API capabilities, allowing detailed inspection of UI elements and their properties. It\'s particularly useful for automating interactions with applications that don\'t have robust AppleScript support or when you need to inspect the UI structure in detail.
216+
217+
**Input Parameters:**
218+
219+
* `command` (enum: 'query' | 'perform', required): The operation to perform.
220+
* `query`: Retrieves information about UI elements.
221+
* `perform`: Executes an action on a UI element (like clicking a button).
222+
223+
* `locator` (object, required): Specifications to find the target element(s).
224+
* `app` (string, required): The application to target, specified by either bundle ID or display name (e.g., "Safari", "com.apple.Safari").
225+
* `role` (string, required): The accessibility role of the target element (e.g., "AXButton", "AXStaticText").
226+
* `match` (object, required): Key-value pairs of attributes to match. Can be empty (`{}`) if not needed.
227+
* `navigation_path_hint` (array of strings, optional): Path to navigate within the application hierarchy (e.g., `["window[1]", "toolbar[1]"]`).
228+
229+
* `return_all_matches` (boolean, optional): When `true`, returns all matching elements rather than just the first match. Default is `false`.
230+
231+
* `attributes_to_query` (array of strings, optional): Specific attributes to query for matched elements. If not provided, common attributes will be included. Examples: `["AXRole", "AXTitle", "AXValue"]`
232+
233+
* `required_action_name` (string, optional): Filter elements to only those supporting a specific action (e.g., "AXPress" for clickable elements).
234+
235+
* `action_to_perform` (string, optional, required when `command="perform"`): The accessibility action to perform on the matched element (e.g., "AXPress" to click a button).
236+
237+
* `report_execution_time` (boolean, optional): If true, the tool will return an additional message containing the formatted script execution time. Defaults to false.
238+
239+
* `limit` (integer, optional): Maximum number of lines to return in the output. Defaults to 500. Output will be truncated if it exceeds this limit.
240+
241+
* `max_elements` (integer, optional): For `return_all_matches: true` queries, this specifies the maximum number of UI elements the `ax` binary will fully process and return attributes for. If omitted, an internal default (e.g., 200) is used. This helps manage performance when querying UIs with a very large number of matching elements (like numerous text fields on a complex web page). This is different from `limit`, which truncates the final text output based on lines.
242+
243+
* `debug_logging` (boolean, optional): If true, enables detailed debug logging from the underlying `ax` binary. This diagnostic information will be included in the response, which can be helpful for troubleshooting complex queries or unexpected behavior. Defaults to false.
244+
245+
* `output_format` (enum: 'smart' | 'verbose' | 'text_content', optional, default: 'smart'): Controls the format and verbosity of the attribute output from the `ax` binary.
246+
* `'smart'`: (Default) Optimized for readability. Omits attributes with empty or placeholder values. Returns key-value pairs.
247+
* `'verbose'`: Maximum detail. Includes all attributes, even empty/placeholders. Key-value pairs. Best for debugging element properties.
248+
* `'text_content'`: Highly compact for text extraction. Returns only concatenated text values of common textual attributes (e.g., AXValue, AXTitle). No keys are returned. Ideal for quickly getting all text from elements; the `attributes_to_query` parameter is ignored in this mode.
249+
250+
**Example Queries (Note: key names have changed to snake_case):**
251+
252+
1. **Find all text elements in the front Safari window:**
253+
```json
254+
{
255+
"command": "query",
256+
"return_all_matches": true,
257+
"locator": {
258+
"app": "Safari",
259+
"role": "AXStaticText",
260+
"match": {},
261+
"navigation_path_hint": ["window[1]"]
262+
}
263+
}
264+
```
265+
266+
2. **Find and click a button with a specific title:**
267+
```json
268+
{
269+
"command": "perform",
270+
"locator": {
271+
"app": "System Settings",
272+
"role": "AXButton",
273+
"match": {"AXTitle": "General"}
274+
},
275+
"action_to_perform": "AXPress"
276+
}
277+
```
278+
279+
3. **Get detailed information about the focused UI element:**
280+
```json
281+
{
282+
"command": "query",
283+
"locator": {
284+
"app": "Mail",
285+
"role": "AXTextField",
286+
"match": {"AXFocused": "true"}
287+
},
288+
"attributes_to_query": ["AXRole", "AXTitle", "AXValue", "AXDescription", "AXHelp", "AXPosition", "AXSize"]
289+
}
290+
```
291+
292+
**Note:** Using this tool requires that the application running this server has the necessary Accessibility permissions in macOS System Settings > Privacy & Security > Accessibility.
293+
211294
## Key Use Cases & Examples
212295

213296
- **Application Control:**

0 commit comments

Comments
 (0)