CONTRIBUTOR-DOCS / Contributor guides / Accessibility testing
In this doc
This guide covers automated accessibility testing for Spectrum Web Components using Playwright. You'll learn how to write, run, and maintain accessibility tests for both 1st-gen and 2nd-gen components.
Automated accessibility testing (ARIA snapshots, aXe-core) catches many issues but cannot capture everything. Manual testing is required because:
- Keyboard behavior — Focus order, tab stops, arrow-key patterns, and whether focus is visible or trapped must be verified by actually using the keyboard. Tools can detect some keyboard-related rules but not the full interaction flow.
- Screen reader behavior — How a component is announced (role, name, state), the order of announcements, and whether state changes are communicated depend on real assistive technology. Automated checks cannot replicate this.
- Context and UX — Whether an interaction makes sense for keyboard-only or screen reader users often requires human judgment.
For every PR that affects interactive components, you must perform manual keyboard and screen reader testing and document your steps in the pull request template. Reviewers use this to ensure accessibility is validated during the review process.
# From project root, 1st-gen, or 2nd-gen directory
yarn test:a11y # Run all tests (both generations)
yarn test:a11y:1st # Run only 1st generation tests
yarn test:a11y:2nd # Run only 2nd generation tests
yarn test:a11y:ui # Interactive UI mode (great for debugging)Tests automatically start the required Storybook instances and run in Chromium.
Two complementary approaches:
Captures the accessibility tree structure and compares it to a baseline. Detects unintentional changes to:
- ARIA roles
- ARIA attributes
- Text content
- Accessibility tree structure
Coverage: ~40% of accessibility issues
Automatically checks ~50+ WCAG 2.0/2.1 Level A/AA rules:
- Color contrast
- Keyboard navigation
- ARIA validity
- Semantic HTML
- Focus management
Coverage: ~50% of accessibility issues
Together: These catch the most common accessibility issues early in development.
Create <component>.a11y.spec.ts in your component's test/ directory:
// 1st-gen/packages/badge/test/badge.a11y.spec.ts
import { expect, test } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import { gotoStory } from '../../../test/a11y-helpers.js';
test.describe('Badge - ARIA Snapshots', () => {
test('should have correct accessibility tree', async ({ page }) => {
const badge = await gotoStory(page, 'badge--default', 'sp-badge');
await expect(badge).toMatchAriaSnapshot();
});
});
test.describe('Badge - aXe Validation', () => {
test('should not have accessibility violations', async ({ page }) => {
await gotoStory(page, 'badge--default', 'sp-badge');
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
.analyze();
expect(results.violations).toEqual([]);
});
});Key details:
- Story ID:
'badge--default'(check Storybook URL atlocalhost:8080) - Element name:
'sp-badge'(the custom element tag name) - Helper import:
'../../../test/a11y-helpers.js'(1st-gen test helpers)
Same pattern, different details:
// 2nd-gen/packages/swc/components/badge/test/badge.a11y.spec.ts
import { expect, test } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import { gotoStory } from '../../../utils/a11y-helpers.js';
test.describe('Badge - ARIA Snapshots', () => {
test('should have correct accessibility tree', async ({ page }) => {
const badge = await gotoStory(
page,
'components-badge--default', // 2nd gen story ID format
'swc-badge' // 2nd gen element name
);
await expect(badge).toMatchAriaSnapshot();
});
});
test.describe('Badge - aXe Validation', () => {
test('should not have accessibility violations', async ({ page }) => {
await gotoStory(page, 'components-badge--default', 'swc-badge');
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
.analyze();
expect(results.violations).toEqual([]);
});
});Key differences:
- Story ID:
'components-badge--default'(check Storybook URL atlocalhost:6006) - Element name:
'swc-badge'(instead ofsp-badge) - Helper import:
'../../../utils/a11y-helpers.js'(from swc utils directory) - Storybook port: 6006 (vs 8080 for 1st gen) - automatically handled by Playwright
Test helpers are available in each generation:
- 1st gen:
1st-gen/test/a11y-helpers.ts - 2nd gen:
2nd-gen/packages/swc/utils/a11y-helpers.ts
Navigate to a Storybook story and wait deterministically for the component to be ready.
Parameters:
page: Playwright Page objectstoryId: Storybook story ID (from the URL)elementSelector: CSS selector for the component (usually the custom element name)
Returns: Playwright Locator for the component
Example:
// Wait for component to be fully ready
const badge = await gotoStory(page, 'badge--default', 'sp-badge');
// Now safe to test
await expect(badge).toMatchAriaSnapshot();How it works:
- Navigates to story URL
- Waits for custom element definition (
customElements.whenDefined) - Waits for Storybook to render content
- Waits for element visibility
- Waits for Web Component upgrade
This eliminates flaky tests caused by testing components before they're ready.
Wait for a custom element to be defined.
await waitForCustomElement(page, 'sp-badge');Wait for Storybook story to render and component to be visible.
const element = await waitForStoryReady(page, 'sp-badge');Open Storybook and navigate to your component. The story ID is in the URL:
1st gen (localhost:8080):
http://localhost:8080/?path=/story/badge--default
^^^^^^^^^^^^^
Story ID
2nd gen (localhost:6006):
http://localhost:6006/?path=/story/components-badge--default
^^^^^^^^^^^^^^^^^^^^^^^^
Story ID
yarn test:a11y # All tests (both generations)
yarn test:a11y:1st # Only 1st generation
yarn test:a11y:2nd # Only 2nd generation
yarn test:a11y:ui # Interactive UI mode# From 1st-gen
cd 1st-gen
yarn test:a11y # All tests (both generations)
yarn test:a11y badge # Specific component
yarn test:a11y:1st # Only 1st gen
yarn test:a11y:2nd # Only 2nd gen
yarn test:a11y badge --update-snapshots # Update ARIA baselines
yarn test:a11y:ui # UI mode
# From 2nd-gen (new home for shared infrastructure)
cd 2nd-gen
yarn test:a11y # All tests (both generations)
yarn test:a11y:1st # Only 1st gen
yarn test:a11y:2nd # Only 2nd gen
yarn test:a11y:ui # UI modeWhen you intentionally change a component's accessibility tree:
yarn test:a11y <component> --update-snapshotsThis updates the baseline ARIA snapshots in <component>.a11y.spec.ts-snapshots/.
ARIA snapshots are saved as YAML files in <test-file>-snapshots/:
# Example: Badge default variant
- text: 'Default'These files are:
- ✅ Committed to git - They're the baseline
- ✅ Updated with
--update-snapshots - ✅ Compared on every run - Detect regressions
When snapshots fail:
- Review the diff in the test output
- If the change is intentional, update snapshots
- If unexpected, fix the component
When aXe finds violations, it reports:
- What's wrong - Rule that failed
- Where it is - Element selector
- How to fix it - Link to documentation
Example:
Expected: []
Received: [
{
id: "color-contrast",
impact: "serious",
description: "Ensures the contrast between foreground and background colors meets WCAG 2 AA",
help: "Elements must have sufficient color contrast",
helpUrl: "https://dequeuniversity.com/rules/axe/4.4/color-contrast",
nodes: [...]
}
]
Do test:
- Default state
- All semantic variants (positive, negative, info, etc.)
- Size variants (s, m, l, xl)
- Interactive states (disabled, selected, focused)
- With different content (text, icons, numbers)
Don't need to test:
- Every color combination
- Every possible prop combination
- Styling details (use visual regression for that)
ARIA snapshot failures:
- Review the diff - is this intentional?
- If yes: Update snapshots with
--update-snapshots - If no: Fix the component to restore expected structure
aXe violations:
- Read the violation message and linked docs
- Fix the component to address the issue
- Re-run tests to verify
Test timeout/hanging:
- Check that Storybook is running
- Verify the story ID is correct
- Ensure the element selector matches
- Start with ARIA snapshots - They're fast to write and catch structural changes
- Add aXe tests for critical paths - Form controls, navigation, overlays
- Use UI mode for debugging -
yarn test:a11y:uishows live browser - Test variants separately - One test per story keeps failures focused
- Commit ARIA snapshots - They're living documentation
playwright.a11y.config.ts (at the root) defines two projects:
projects: [
{
name: '1st-gen',
testMatch: '**/packages/*/test/**/*.a11y.spec.ts',
use: { baseURL: 'http://localhost:8080' },
},
{
name: '2nd-gen',
testMatch: '**/packages/swc/components/*/test/**/*.a11y.spec.ts',
use: { baseURL: 'http://localhost:6006' },
},
];This allows both generations to run against their respective Storybook instances.
Tests automatically start Storybook when needed:
webServer: [
{
command: 'cd ../1st-gen && yarn storybook',
port: 8080,
reuseExistingServer: !process.env.CI,
},
{
command: 'cd packages/swc && yarn storybook',
port: 6006,
reuseExistingServer: !process.env.CI,
},
];spectrum-web-components/
├── playwright.a11y.config.ts # Playwright config (both gens)
├── CONTRIBUTOR-DOCS/
│ └── 01_contributor-guides/
│ └── 09_accessibility-testing.md # This guide
├── 1st-gen/
│ ├── package.json # Test scripts (points to root config)
│ ├── test/
│ │ └── a11y-helpers.ts # 1st gen test helpers
│ └── packages/
│ ├── badge/test/
│ │ ├── badge.a11y.spec.ts # Tests
│ │ └── badge.a11y.spec.ts-snapshots/ # ARIA baselines
│ └── status-light/test/
│ ├── status-light.a11y.spec.ts
│ └── status-light.a11y.spec.ts-snapshots/
└── 2nd-gen/
├── package.json # Test scripts (points to root config)
└── packages/swc/
├── utils/
│ └── a11y-helpers.ts # 2nd gen test helpers
└── components/
├── badge/test/
│ ├── badge.a11y.spec.ts
│ └── badge.a11y.spec.ts-snapshots/
└── status-light/test/
├── status-light.a11y.spec.ts
└── status-light.a11y.spec.ts-snapshots/
In addition to automated tests, contributors must complete the Accessibility testing checklist in the pull request template and document their keyboard and screen reader testing steps. The following resources help you perform and document that testing.
When testing with the keyboard, verify and document:
- Tab order — Tab moves focus into and through the component in a logical order; no focus traps.
- Activation — Enter and/or Space activate buttons, links, and controls as expected.
- Arrow keys — For components that use arrow keys (tabs, menus, sliders, listboxes, etc.), ↑ ↓ ← → move focus or change value as per the WAI-ARIA keyboard interface.
- Escape — Escape dismisses overlays, popovers, or menus when applicable.
- Focus visibility — Focus indicator is clearly visible and not obscured.
Reference: WAI-ARIA Authoring Practices Guide — Keyboard interface and APG patterns for component-specific keyboard patterns.
In the PR template, fill in: where you tested (e.g. Storybook URL), what you did (e.g. "Tabbed to the menu, used arrow keys to move between items"), and the expected result.
When testing with a screen reader, verify and document:
- Role and name — The component is announced with the correct role and an understandable name (or label).
- State — States such as expanded/collapsed, selected, checked, and disabled are announced when they change.
- Relationships — Labels, descriptions, and group relationships are announced so the structure is clear.
- No clutter — There are no unnecessary or duplicate announcements.
Tools: Test with at least one screen reader, for example:
- Windows: NVDA (free) or JAWS
- macOS: VoiceOver (built-in; Cmd+F5 to toggle)
- Mobile: VoiceOver (iOS), TalkBack (Android)
In the PR template, document where you tested, what you did (e.g. "Navigated to the menu with VO and arrow keys, opened it with VO+Space"), and the expected announcements or result.
For developers:
- Catch issues in seconds, not days
- Clear, actionable failure messages
- No manual testing needed for basic checks
For the project:
- Scalable to all components
- CI-ready (runs on every PR)
- Complements manual testing