Skip to content

[SPIKE] Research about Computer-Use, how feasible it is to implement using Go or C binding #235

@edenreich

Description

@edenreich

Summary

Investigate how viable it is to make a CLI program also able to control the computer using Computer-Use tools (like mouse click, mouse move, take screenshot etc).

Since images are expensive in token we need to come up with a solution to "denoising" images and reducing their size to include only that's what is essential.

Only subset of the LLMs will support this.

Acceptance Criteria

  • There is a clear path for implementing Computer-Use
  • Tasks are broken down into multiple actionable Github Issues
  • Security is considered, for every click action on the first phase there is a need for approval window

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions