[Extending API] /screenshot

[raiden-staging]

• Extending server with /screenshot endpoint

# Screenshot

# | GET /screenshot
curl http://localhost:10001/screenshot -o screenshot.png

# | POST /screenshot
curl -X POST -H "Content-Type: application/json" --data '{"x":0,"y":0,"width":200,"height":200}' -o screenshot_region.png http://localhost:10001/screenshot

---

TL;DR

Added new /screenshot API endpoints to capture full or partial screen images, and an uptime health check.

Why we made these changes

To extend the server's API capabilities, providing programmatic screenshot capture for integration or automation, and to expose server health and uptime information.

What changed?

• server/cmd/api/api/screenshot.go: New file introducing GET /screenshot for full-screen captures and POST /screenshot for region-specific captures, leveraging ffmpeg.
• server/cmd/api/api/api.go: Implemented server uptime tracking and exposed it via a new GetHealth endpoint.
• server/README.md & server/openapi.yaml: Updated documentation to reflect the new /screenshot API routes and their usage.
• server/cmd/api/api/screenshot_test.go: Added comprehensive unit tests for the health check and screenshot capture functionalities, including input validation.
• server/go.mod & server/go.sum: Updated Go module dependencies.

^{Description generated by Mesa. Update settings}

[mesa-dot-dev]

Performed full review of fe02e69...0377a23

^{7 files reviewed | 0 comments | Review on Mesa | Edit Reviewer Settings}

[cursor]

Bug: Screen Dimensions Fallback Hides Errors

The getScreenDimensions function returns a nil error even when xdpyinfo fails, causing it to silently use fallback dimensions. This prevents captureScreenshot from logging a warning and leads to incorrect bounds validation, potentially rejecting valid screenshot regions or allowing invalid ones that cause ffmpeg to fail.

[matthewjmarangoni]

This feature will be nice to use!

1. At least one agent uses a base 64 encoded PNG (see Computer Use). Making a base 64 encoding format available would be useful as users could relay it.
2. Should the screenshot endpoint be under /screen to group it with other screen operations?
3. Instead of separating the region capture by different actions consider using an optional parameter for the POST action or a separate endpoint.

[matthewjmarangoni]

If the fallback is needed would it be useful to read fallback dimensions from something such as environment variables before fixed values?

[rgarcia]

thanks for this @raiden-staging !

[rgarcia]

this doesn't look right

[raiden-staging]

oapi/go version conflicts fix induced slop 🙃 sorry for the mess will fix

[rgarcia]

I think I'd prefer merging these into one POST request

[rgarcia]

not sure how oapi does response body reading under the hood but I'd assume it's doing this read of the reader anyway, so this feels like duplicate / unnecessary reading of the screenshot into memory

[raiden-staging]

should we serve as base64 string [ @matthewjmarangoni ] or binary (or add optional ?format={} ) ?

[matthewjmarangoni]

If I understand the question correctly, offering both with a way for the client to choose either the original binary or base 64 encoded string is what I had in mind.

[rgarcia]

thanks again for this @raiden-staging -- screenshot functionality has been added as part of #76