Skip to content

[FEATURE] Add tools for computer use #358

@edenreich

Description

@edenreich

Summary

Currently this CLI only support non-gui applications.
The goal of computer use is to actually let the AI reason about denoised and compressed screenshots, click things, open apps, type etc.

In a GUI world it's getting more expensive so it's important to ensure a compression and denoising is done properly to avoid excessive usage of tokens.

For the first version it's supports only Linux Ubuntu. Windows or Mac-OS is out of scope (future versions).

Acceptance Criteria

  • There is a tool set of computer use as a separate config section.
  • Computer use is disabled by default.
  • The web terminal streaming the UI of the remote machine in a window - similar to how vnc works.
  • It's documented.
  • It's tested.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions