feat: Add computer use tools for remote and local GUI automation #359
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Rename screenshot streaming UI components to use "Preview" terminology: - Rename screenshot-overlay.js to preview-overlay.js - Remove emoji from button (📷 Screenshots → Preview) - Update overlay title from "Live Screenshot" to "Live Preview" - Update user-facing messages to use "Preview" instead of "Screenshot" This improves clarity and consistency in the web UI while keeping internal implementation details (CSS classes, API endpoints) unchanged. Signed-off-by: Eden Reich <[email protected]>
Signed-off-by: Eden Reich <[email protected]>
Contributor
Author
TODOs
|
edenreich
changed the title
Signed-off-by: Eden Reich <[email protected]>
Signed-off-by: Eden Reich <[email protected]>
Signed-off-by: Eden Reich <[email protected]>
**Thread Safety & Race Conditions:** - Add thread-safe WindowCoordinator with serial DispatchQueue - Protect window arrays (borders, click indicators, move trails) from concurrent access - Fix segmentation faults during computer use tool execution - Add process safety checks in writeEvent to prevent writes to dead processes **Coordinate System Fixes:** - Fix double coordinate conversion in click/move indicators - Remove redundant Y-axis flip in Swift (Go already converts to macOS coords) - Click indicators and move trails now appear at correct screen positions **Control Event Architecture:** - Add control event forwarder for GUI → TUI communication - Implement dedicated always-open channel for pause/resume events - Revert EventBridge.Tap() to simple unidirectional design - Add GetEventBridge() to StateManager interface **Code Organization:** - Extract view classes into separate files: - ClickIndicator.swift: Circular ring indicator - MoveTrail.swift: Arrow showing mouse movement - ControlBar.swift: Pause/resume button bar - ImageThumbnail.swift: Full-screen image viewer - Refactor monitorProcess into respawnWindow and restoreBorderOverlay - Update build.sh to include all view files **Manual Tool Execution Fixes:** - Fix completion event status: "complete" → "completed" - Add image attachments to completion events - GetLatestScreenshot now properly shows completion and displays images - Manual tools (!! syntax) now broadcast events to floating window **Other Improvements:** - Remove dead code in screenshot_server.go - Simplify chat handler resume logic - Clean up verbose comments in manager.go Fixes segfaults during MouseClick/MouseMove operations and ensures all visual indicators work correctly with proper event completion. Signed-off-by: Eden Reich <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds computer use capabilities including screenshot capture, mouse movement, mouse clicks, and keyboard typing. Includes a complete Docker-based example with Ubuntu GUI desktop, X11/Wayland support, and live screenshot streaming to web UI. Closes #358.
Key features:
Technical details: