Skip to content

Add container visualization for accessibility snapshots#278

Draft
RoyalPineapple wants to merge 8 commits intomainfrom
a11y-container-graph
Draft

Add container visualization for accessibility snapshots#278
RoyalPineapple wants to merge 8 commits intomainfrom
a11y-container-graph

Conversation

@RoyalPineapple
Copy link
Copy Markdown
Collaborator

@RoyalPineapple RoyalPineapple commented Dec 2, 2025

Adds container visualization to accessibility snapshot legends without changing element overlay behavior.

Summary

  • New showContainers configuration flag turns on a hierarchical legend that groups elements by their accessibility container with dashed borders and per-container badges
  • Snapshot image and element overlays are baked into a single flat UIImage before the legend enters layout, so overlay positions can never be shifted by legend placement
  • Parser/render order swap fixes a pre-existing overlay misalignment for nested UIHostingController content (e.g. SemanticGroupWrapper)
  • Demo coverage for every container shape the parser produces (semanticGroup × 4 info variants, list, landmark, dataTable, tabBar) plus a SwiftUI Table reveal
  • iOS 26.4 reference images recorded for the new tests

Details

Snapshot + overlay baking

PreParsedAccessibilitySnapshotView.init composites the snapshot image and all element overlays into a single flat UIImage up front. The rendered snapshotWithOverlays body is just Image(uiImage:) — no ZStack of overlays on top of the image. This decouples overlay alignment from the surrounding SwiftUI layout, so the legend (right, below, multi-column, future flow layouts) can never shift overlay coordinates.

Baking uses drawHierarchy(afterScreenUpdates: true) inside a temporary UIWindow with safeAreaRegions = [] so SwiftUI content renders correctly without safe-area offsets.

Overlay alignment fix

drawHierarchy(afterScreenUpdates: true) mutates layout mid-render for views that defer layout until the render cycle — notably nested UIHostingController instances inside UIViewRepresentable wrappers (e.g. SemanticGroupWrapper). When parsing happened after rendering, the parser read accessibilityFrame values from a post-render layout that no longer matched what the image captured, shifting the first group's overlay.

Fix: swap the parse/render order in AccessibilitySnapshotBaseView.parseAccessibility() so marker frames and the rendered image both come from the same pre-render layout state.

Container legend

When showContainers is enabled, the legend renders a hierarchical view that groups elements by their accessibility containers using dashed borders and container badges. Container entries expand to the full legend column width so multi-word badges like "Data Table (3 × 4)" stay on one line. The snapshot area is identical regardless of showContainers — only the legend content changes.

Supported container types in the badge label:

  • semanticGroup(label, value, identifier) → "label: value", or identifier, or "Semantic Group"
  • list → "List"
  • landmark → "Landmark"
  • dataTable(rows, cols) → "Data Table (r × c)"
  • tabBar → "Tab Bar"

New types:

  • HierarchyColorAssignment — assigns color indices to hierarchy nodes (elements use traversal-order position matching overlay indices; containers use a separate counter)
  • ContainerLegendEntryView / ContainerBadge — dashed-border legend entry with layered-icon badge
  • HierarchyLegendView — recursive hierarchical legend renderer

Demos and tests

Each container type now has a focused demo + test under AccessibilitySnapshotPreviewsDemo:

  • ContainerDemo — four semanticGroup info variants (label only, label + value, identifier only, all three)
  • ListContainerDemoaccessibilityContainerType = .list
  • LandmarkContainerDemoaccessibilityContainerType = .landmark
  • TabBarContainerDemoaccessibilityTraits = .tabBar on a non-UITabBar view
  • DataTableContainerDemoaccessibilityContainerType = .dataTable with real UIAccessibilityContainerDataTableCell elements reporting per-cell row/column ranges, so the legend shows "Data Table (3 × 4)" plus each cell's row/column context
  • SwiftUITableDemo — exposes how SwiftUI's native Table collapses to a one-column List on iPhone; the parser still emits a .dataTable container but with rowCount = 0, columnCount = 0 and no UIAccessibilityContainerDataTableCell elements

Shared AccessibilityContainers.swift provides:

  • A generic UIViewRepresentable that wraps SwiftUI content in a UIView reporting a configured UIAccessibilityContainerType or the .tabBar trait
  • An AccessibilityDataTable UIViewRepresentable that renders a 2D grid of labels and exposes each cell as a real UIAccessibilityElement conforming to UIAccessibilityContainerDataTableCell

Other changes

  • FBSnapshotTestCase+SwiftUI: sets hostingController.safeAreaRegions = [] on iOS 16.4+ in the config-based overload, preventing drawHierarchy from baking status bar / safe-area offsets into the snapshot
  • AccessibilitySnapshotConfiguration: new showContainers: Bool = false flag
  • ParsedAccessibilityData: now also carries the full hierarchy: [AccessibilityHierarchy] tree (not just flattened markers) so the container legend can render

Test plan

  • All existing snapshot tests pass
  • Six new container-demo tests produce overlays aligned 1:1 with snapshot content
  • Container legend correctly groups elements with dashed borders and full-width badges
  • DataTable demo reports per-cell row/column context (no NSNotFound noise)
  • iOS 26.4 reference images recorded for all new tests

🤖 Generated with Claude Code

@RoyalPineapple RoyalPineapple changed the title Add accessibility container visualization with hierarchy support [WIP] Add accessibility container visualization with hierarchy support Dec 3, 2025
@RoyalPineapple RoyalPineapple force-pushed the a11y-container-graph branch 3 times, most recently from 3d8c515 to 3733a88 Compare December 3, 2025 16:40
@RoyalPineapple RoyalPineapple changed the title [WIP] Add accessibility container visualization with hierarchy support Add hierarchical accessibility parsing API Dec 4, 2025
@RoyalPineapple RoyalPineapple force-pushed the a11y-container-graph branch 7 times, most recently from 452d709 to b531faf Compare December 9, 2025 12:45
Copy link
Copy Markdown
Collaborator Author

@RoyalPineapple RoyalPineapple left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still in progress

@RoyalPineapple RoyalPineapple force-pushed the a11y-container-graph branch 9 times, most recently from e2c65ec to bb6609f Compare January 28, 2026 17:21
@RoyalPineapple RoyalPineapple changed the title Add hierarchical accessibility parsing API Add container visualization for accessibility snapshots Jan 28, 2026
@RoyalPineapple RoyalPineapple changed the base branch from main to a11y-hierarchy-parsing January 28, 2026 17:22
@RoyalPineapple RoyalPineapple changed the base branch from a11y-hierarchy-parsing to main January 28, 2026 17:28
RoyalPineapple and others added 4 commits April 29, 2026 16:02
When showContainers is enabled in the configuration, the legend renders
a hierarchical view that groups elements by their accessibility
containers using dashed borders and container badges.

Scope: legend-only. All element overlay rendering (ElementOverlay,
positions, colors, coordinates) is completely unchanged. The snapshot
image and its overlays are rendered identically regardless of
showContainers — only the legend content differs.

New files:
- HierarchyColorAssignment: assigns color indices to hierarchy nodes
  (elements use traversal-order position to match overlay colors;
  containers use a separate counter)
- ContainerLegendEntryView: dashed-border legend entry with a badge
  that displays the container type (Semantic Group/List/Landmark/
  Data Table/Tab Bar) or its label/value/identifier
- HierarchyLegendView: recursive hierarchical legend renderer
- ContainerDemo: demo using SemanticGroupWrapper to create real UIKit
  accessibility containers from a SwiftUI demo (SwiftUI has no native
  equivalent for UIAccessibilityContainerType.semanticGroup)

Changes:
- AccessibilitySnapshotConfiguration: add showContainers flag
- ParsedAccessibilityData: add hierarchy tree alongside flat markers
- PreParsedAccessibilitySnapshotView: swap legend based on showContainers;
  snapshot+overlay section untouched
- SwiftUIAccessibilitySnapshotContainerView: pass hierarchy through

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Composites the snapshot image with its element overlays into a single
UIImage inside PreParsedAccessibilitySnapshotView, so overlay coordinates
are locked to the snapshot pixels before the legend participates in the
surrounding SwiftUI layout. This guarantees a 1:1 mapping between the
snapshot and its overlays regardless of legend placement.

Also swaps parse/render order in AccessibilitySnapshotBaseView so marker
frames and the rendered image are taken from the same pre-render layout
state. drawHierarchy(afterScreenUpdates: true) mutates layout for nested
UIHostingControllers inside UIViewRepresentable wrappers (e.g. the
SemanticGroupWrapper demo), which previously caused the first group's
overlay to shift relative to the captured snapshot.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Splits the single ContainerDemo into a focused set of demo views that each
exercise one container shape the parser produces:

- ContainerDemo (updated) — four semanticGroup info variants:
  label only, label + value, identifier only, and label + value + id
- ListContainerDemo — accessibilityContainerType = .list
- LandmarkContainerDemo — accessibilityContainerType = .landmark
- TabBarContainerDemo — accessibilityTraits = .tabBar on a non-UITabBar view
- DataTableContainerDemo — accessibilityContainerType = .dataTable with real
  UIAccessibilityContainerDataTableCell elements reporting per-cell row and
  column ranges, so the legend shows "Data Table (r × c)" and each cell's
  row/column context rather than NSNotFound noise

Shared AccessibilityContainers.swift provides:
- A generic UIViewRepresentable that wraps SwiftUI content in a UIView
  reporting a configured UIAccessibilityContainerType or the .tabBar trait
- An AccessibilityDataTable UIViewRepresentable that renders a 2D grid of
  labels and exposes each cell as a real UIAccessibilityElement conforming
  to UIAccessibilityContainerDataTableCell

One test per new demo (all with showContainers: true) plus iOS 26.4
reference images. testContainerDemo/testContainerDemoWithoutContainers
references re-recorded for the new semanticGroup variants layout.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SwiftUITableDemo exercises native SwiftUI Table accessibility on iPhone:
the parser does see accessibilityContainerType = .dataTable, but Table
collapses to a one-column List on iPhone and reports rowCount = 0,
columnCount = 0 with no UIAccessibilityContainerDataTableCell elements,
so the legend shows "Data Table (0 × 0)".

Container legend entries now expand to the full available width
(maxWidth: .infinity) so badges like "Data Table (0 × 0)" sit on a
single line instead of wrapping to fit the narrowest child.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The outer hostingController.safeAreaRegions = [] flipped the rendering
of every SwiftUI snapshot in the project, breaking testSimpleView and
both SwiftUIListSectionTests across iOS 17/18/26 because their reference
images were recorded with the safe area inset on. The PreParsedAccessibilitySnapshotView
bake step has its own internal safeAreaRegions = [] for compositing,
so removing the outer one has no effect on the new container tests'
correctness — only their bytes shift slightly, hence the rerecord.
The previous fix removed `safeAreaRegions = []` from every SwiftUI snapshot,
breaking the AccessibilitySnapshotPreviewsTests which expect edge-to-edge
rendering through the new bake pipeline. Scope the opt-out to the SwiftUI
layout engine so the legacy UIKit engine keeps the safe area inset its
references were recorded with.

Re-record the affected references:
- iOS 18.5: all 13 SwiftUIRendererTests now match the bake-step output
- iOS 26.4: all 13 SwiftUIRendererTests, including refs that didn't exist
  before for the 6 pre-existing tests
- iOS 17.5: testTabBars rebaselined for sub-pixel drift introduced by the
  parse-before-render reorder
Sub-pixel rendering differs between local Macs and the GitHub Actions
runners, so references recorded locally don't match what CI produces.
Replace the affected references with the actual captured images from the
previous CI run's Failed Image attachments. For the 7 new container tests
on iOS 26.2, seed placeholders (copies of the 26.4 references) so CI
actually renders them on the next run instead of bailing with "Reference
image not found"; the resulting Failed Images can replace these placeholders.
The previous commit seeded placeholders (copies of the iOS 26.4 references)
for 7 tests so CI would render them and emit Failed Image attachments.
This commit swaps those in for the actual iOS 26.2 captures from CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant