|
2 | 2 |
|
3 | 3 | [](https://github.com/OpenAdaptAI/openadapt-capture) |
4 | 4 |
|
5 | | -> *Auto-generated from [OpenAdaptAI/openadapt-capture](https://github.com/OpenAdaptAI/openadapt-capture). Last synced: 2026-03-03 21:03 UTC* |
| 5 | +> *Auto-generated from [OpenAdaptAI/openadapt-capture](https://github.com/OpenAdaptAI/openadapt-capture). Last synced: 2026-03-03 21:32 UTC* |
6 | 6 |
|
7 | 7 | --- |
8 | 8 |
|
9 | | -# OpenAdapt Capture |
| 9 | +# openadapt-example |
10 | 10 |
|
11 | | -[](https://github.com/OpenAdaptAI/openadapt-capture/actions/workflows/test.yml) |
12 | | -[](https://opensource.org/licenses/MIT) |
13 | | -[](https://www.python.org/downloads/) |
14 | | - |
15 | | -[](https://pypi.org/project/openadapt-capture/) |
16 | | -[](https://pypi.org/project/openadapt-capture/) |
17 | | - |
18 | | -**OpenAdapt Capture** is the data collection component of the [OpenAdapt](https://github.com/OpenAdaptAI) GUI automation ecosystem. |
19 | | - |
20 | | -Capture platform-agnostic GUI interaction streams with time-aligned screenshots and audio for training ML models or replaying workflows. |
21 | | - |
22 | | -> **Status:** Pre-alpha. |
23 | | -
|
24 | | ---- |
25 | | - |
26 | | -## The OpenAdapt Ecosystem |
27 | | - |
28 | | -``` |
29 | | - OpenAdapt GUI Automation Pipeline |
30 | | - ================================= |
31 | | -
|
32 | | - +-----------------+ +------------------+ +------------------+ |
33 | | - | | | | | | |
34 | | - | openadapt- | ------> | openadapt-ml | ------> | Deploy | |
35 | | - | capture | Convert | (Train & Eval) | Export | (Inference) | |
36 | | - | | | | | | |
37 | | - +-----------------+ +------------------+ +------------------+ |
38 | | - | | | |
39 | | - v v v |
40 | | - - Record GUI - Fine-tune VLMs - Run trained |
41 | | - interactions - Evaluate on agent on new |
42 | | - - Mouse, keyboard, benchmarks (WAA) tasks |
43 | | - screen, audio - Compare models - Real-time |
44 | | - - Privacy scrubbing - Cloud GPU training automation |
45 | | -
|
46 | | -``` |
47 | | - |
48 | | -| Component | Purpose | Repository | |
49 | | -|-----------|---------|------------| |
50 | | -| **openadapt-capture** | Record human demonstrations | [GitHub](https://github.com/OpenAdaptAI/openadapt-capture) | |
51 | | -| **openadapt-ml** | Train and evaluate GUI automation models | [GitHub](https://github.com/OpenAdaptAI/openadapt-ml) | |
52 | | -| **openadapt-privacy** | PII scrubbing for recordings | [GitHub](https://github.com/OpenAdaptAI/openadapt-privacy) | |
53 | | - |
54 | | ---- |
| 11 | +A sample package for testing documentation generation. |
55 | 12 |
|
56 | 13 | ## Installation |
57 | 14 |
|
58 | 15 | ```bash |
59 | | -uv add openadapt-capture |
| 16 | +pip install openadapt-example |
60 | 17 | ``` |
61 | 18 |
|
62 | | -This includes everything needed to capture and replay GUI interactions (mouse, keyboard, screen recording). |
63 | | - |
64 | | -For audio capture with Whisper transcription (large download): |
65 | | - |
66 | | -```bash |
67 | | -uv add "openadapt-capture[audio]" |
68 | | -``` |
69 | | - |
70 | | -## Quick Start |
71 | | - |
72 | | -### Capture |
| 19 | +## Usage |
73 | 20 |
|
74 | 21 | ```python |
75 | | -from openadapt_capture import Recorder |
76 | | - |
77 | | -# Record GUI interactions |
78 | | -with Recorder("./my_capture", task_description="Demo task") as recorder: |
79 | | - # Captures mouse, keyboard, and screen until context exits |
80 | | - input("Press Enter to stop recording...") |
| 22 | +from openadapt_example import run |
| 23 | +run() |
81 | 24 | ``` |
82 | 25 |
|
83 | | -### Replay / Analysis |
84 | | - |
85 | | -```python |
86 | | -from openadapt_capture import Capture |
87 | | - |
88 | | -# Load and iterate over time-aligned events |
89 | | -capture = Capture.load("./my_capture") |
90 | | - |
91 | | -for action in capture.actions(): |
92 | | - # Each action has an associated screenshot |
93 | | - print(f"{action.timestamp}: {action.type} at ({action.x}, {action.y})") |
94 | | - screenshot = action.screenshot # PIL Image at time of action |
95 | | -``` |
96 | | - |
97 | | -### Low-Level API |
98 | | - |
99 | | -```python |
100 | | -from openadapt_capture.db import create_db, get_session_for_path |
101 | | -from openadapt_capture.db import crud |
102 | | -from openadapt_capture.db.models import Recording, ActionEvent |
103 | | - |
104 | | -# Create a database |
105 | | -engine, Session = create_db("/path/to/recording.db") |
106 | | -session = Session() |
107 | | - |
108 | | -# Insert a recording |
109 | | -recording = crud.insert_recording(session, { |
110 | | - "timestamp": 1700000000.0, |
111 | | - "monitor_width": 1920, |
112 | | - "monitor_height": 1080, |
113 | | - "platform": "win32", |
114 | | - "task_description": "My task", |
115 | | -}) |
116 | | - |
117 | | -# Insert events |
118 | | -crud.insert_action_event(session, recording, 1700000001.0, { |
119 | | - "name": "click", |
120 | | - "mouse_x": 100.0, |
121 | | - "mouse_y": 200.0, |
122 | | - "mouse_button_name": "left", |
123 | | - "mouse_pressed": True, |
124 | | -}) |
125 | | - |
126 | | -# Query events back |
127 | | -from openadapt_capture.capture import CaptureSession |
128 | | -capture = CaptureSession.load("/path/to/capture_dir") |
129 | | -actions = list(capture.actions()) |
130 | | -``` |
131 | | - |
132 | | -## Event Types |
133 | | - |
134 | | -**Raw events** (captured): |
135 | | -- `mouse.move`, `mouse.down`, `mouse.up`, `mouse.scroll` |
136 | | -- `key.down`, `key.up` |
137 | | - |
138 | | -**Actions** (processed): |
139 | | -- `mouse.singleclick`, `mouse.doubleclick`, `mouse.drag` |
140 | | -- `key.type` (merged keystrokes into text) |
141 | | - |
142 | | -## Architecture |
143 | | - |
144 | | -The recorder uses a multi-process architecture copied from legacy OpenAdapt: |
145 | | - |
146 | | -- **Reader threads**: Capture mouse, keyboard, screen, and window events into a central queue |
147 | | -- **Processor thread**: Routes events to type-specific write queues |
148 | | -- **Writer processes**: Persist events to SQLAlchemy DB (one process per event type) |
149 | | -- **Action-gated video**: Only encodes video frames when user actions occur |
150 | | - |
151 | | -``` |
152 | | -capture_directory/ |
153 | | -├── recording.db # SQLite: events, screenshots, window events, perf stats |
154 | | -├── oa_recording-{ts}.mp4 # Screen recording (action-gated) |
155 | | -└── audio.flac # Audio (optional) |
156 | | -``` |
157 | | - |
158 | | -## Performance Testing |
159 | | - |
160 | | -Run a performance test with synthetic input: |
161 | | - |
162 | | -```bash |
163 | | -uv run python scripts/perf_test.py |
164 | | -``` |
165 | | - |
166 | | -This records for 10 seconds using pynput Controllers, then reports: |
167 | | -- Wall/CPU time and memory usage |
168 | | -- Event counts and action types |
169 | | -- Output file sizes |
170 | | -- Memory usage plot (saved to capture directory) |
171 | | - |
172 | | -Run integration tests (requires accessibility permissions): |
173 | | - |
174 | | -```bash |
175 | | -uv run pytest tests/test_performance.py -v -m slow |
176 | | -``` |
177 | | - |
178 | | -## Visualization |
179 | | - |
180 | | -Generate animated demos and interactive viewers from recordings: |
181 | | - |
182 | | -### Animated GIF Demo |
183 | | - |
184 | | -```python |
185 | | -from openadapt_capture import Capture, create_demo |
186 | | - |
187 | | -capture = Capture.load("./my_capture") |
188 | | -create_demo(capture, output="demo.gif", fps=10, max_duration=15) |
189 | | -``` |
190 | | - |
191 | | -### Interactive HTML Viewer |
192 | | - |
193 | | -```python |
194 | | -from openadapt_capture import Capture, create_html |
195 | | - |
196 | | -capture = Capture.load("./my_capture") |
197 | | -create_html(capture, output="viewer.html", include_audio=True) |
198 | | -``` |
199 | | - |
200 | | -## Sharing Recordings |
201 | | - |
202 | | -Share recordings between machines using [Magic Wormhole](https://magic-wormhole.readthedocs.io/): |
203 | | - |
204 | | -```bash |
205 | | -# On the sending machine |
206 | | -capture share send ./my_capture |
207 | | -# Shows a code like: 7-guitarist-revenge |
208 | | - |
209 | | -# On the receiving machine |
210 | | -capture share receive 7-guitarist-revenge |
211 | | -``` |
212 | | - |
213 | | -The `share` command compresses the recording, sends it via Magic Wormhole, and extracts it on the receiving end. No account or setup required - just share the code. |
214 | | - |
215 | | -## Optional Extras |
216 | | - |
217 | | -| Extra | Features | |
218 | | -|-------|----------| |
219 | | -| `audio` | Audio capture + Whisper transcription | |
220 | | -| `privacy` | PII scrubbing ([openadapt-privacy](https://github.com/OpenAdaptAI/openadapt-privacy)) | |
221 | | -| `share` | Recording sharing via Magic Wormhole | |
222 | | -| `all` | Everything | |
223 | | - |
224 | | ---- |
225 | | - |
226 | | -## Development |
227 | | - |
228 | | -```bash |
229 | | -uv sync --dev |
230 | | -uv run pytest tests/ -v --ignore=tests/test_browser_bridge.py |
231 | | - |
232 | | -# Run slow integration tests (requires accessibility permissions) |
233 | | -uv run pytest tests/ -v -m slow |
234 | | -``` |
235 | | - |
236 | | -## Related Projects |
237 | | - |
238 | | -- [openadapt-ml](https://github.com/OpenAdaptAI/openadapt-ml) - Train and evaluate GUI automation models |
239 | | -- [openadapt-privacy](https://github.com/OpenAdaptAI/openadapt-privacy) - PII detection and scrubbing for recordings |
240 | | -- [openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals) - Benchmark evaluation for GUI agents |
241 | | -- [Windows Agent Arena](https://github.com/microsoft/WindowsAgentArena) - Benchmark for Windows GUI agents |
242 | | - |
243 | | -## License |
| 26 | +## Features |
244 | 27 |
|
245 | | -MIT |
| 28 | +- Feature one |
| 29 | +- Feature two |
| 30 | +- Feature three |
246 | 31 |
|
247 | 32 |
|
248 | 33 | --- |
|
0 commit comments