[feat]: add coordinate-space diagnostics for screenshot, AX, and tap workflows

[BigCactusLabs]

The Pain

While manually verifying native iOS UI interactions on iPhone 17 / iOS Simulator 26.4, Baguette could reliably tap lower controls and content-area controls, but an attempted top-toolbar tap did not activate the expected control. I do not have enough evidence to call that a Baguette hit-testing bug. The stronger finding is that the current CLI workflow leaves multiple nearby coordinate spaces easy to mix up.

On the same iPhone 17 simulator:

baguette screenshot:    1206 x 2622 framebuffer pixels
baguette describe-ui:   402 x 874 AX/device-point root
baguette chrome layout: 400 x 872 screen inside 454 x 908 chrome composite

tap expects device points and describe-ui reports device-point frames, while screenshot returns framebuffer pixels and chrome layout describes a chrome composite. The docs mention these concepts, but the CLI does not currently give agents an operational, copy-paste-safe path from "I found this element or pixel" to "this is the canonical tap envelope."

This is especially risky for small controls near the top safe area/status/navigation region, where a small unit or offset mismatch can miss the intended target while lower controls still appear to work.

The Fix

Please add coordinate-space diagnostics and/or docs that make the screenshot -> AX -> tap workflow unambiguous.

Useful outcomes would include one or more of:

# Given an AX hit-test point or node, print the exact tap envelope.
baguette describe-ui --udid <UDID> --x 320 --y 52 --tap-envelope

# Given screenshot pixels, convert to device-point tap coordinates.
baguette coordinates convert --udid <UDID> --from framebuffer --x 960 --y 156

# Dump the active coordinate spaces together.
baguette coordinates describe --udid <UDID>

Example diagnostic output:

{
  "devicePoints": {"width": 402, "height": 874},
  "framebufferPixels": {"width": 1206, "height": 2622, "scale": 3},
  "chromeComposite": {"width": 454, "height": 908},
  "chromeScreen": {"x": 27, "y": 18, "width": 400, "height": 872},
  "tapEnvelope": {"type": "tap", "x": 320, "y": 52, "width": 402, "height": 874}
}

Before & After

Before: an agent using screenshots, AX frames, and chrome layout has to infer which width/height pair belongs in a tap envelope and how to convert screenshot pixels back to device points.

After: Baguette can either emit the canonical tap envelope directly or document the conversion workflow with enough precision that agents can safely target small controls near safe areas.

Acceptance Criteria

• Documentation clearly distinguishes framebuffer pixels, AX/device points, and chrome composite/screen coordinates.
• Documentation includes a worked iPhone example converting a screenshot pixel coordinate into a tap envelope.
• Documentation explains that describe-ui frame centers can be piped directly into tap with the correct point-space width/height.
• A CLI diagnostic or helper emits the active simulator coordinate-space dimensions in one place.
• If feasible, describe-ui can emit a canonical tap envelope for a hit-tested node or selected node center.

Environment / Evidence

• Baguette observed version: v0.1.70 from Homebrew.
• Latest upstream release checked: v0.1.71, released 2026-05-12.
• Simulator: iPhone 17, iOS Simulator 26.4.
• Observed dimensions:
  • baguette screenshot --output ...: 1206x2622 JPEG.
  • baguette describe-ui: root frame 402x874.
  • baguette chrome layout: screen.width=400, screen.height=872, screen.x=27, screen.y=18, composite.width=454, composite.height=908.