Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
fd7e42c
fix link
marktnoonan Oct 18, 2024
2b467be
initial pass at guide
marktnoonan Oct 18, 2024
70ce39c
fix missing label
marktnoonan Oct 18, 2024
47c711a
various text updates
marktnoonan Oct 18, 2024
5152c28
cleanup and clarity
marktnoonan Oct 18, 2024
db9fe83
cleanup and clarity
marktnoonan Oct 18, 2024
a94c9d6
cleanup and clarity
marktnoonan Oct 18, 2024
ecad352
cleanup and clarity
marktnoonan Oct 18, 2024
8a11e6a
remove quotes
marktnoonan Oct 18, 2024
f1129a7
lint fix
marktnoonan Oct 18, 2024
19f9180
cleanup and clarity
marktnoonan Oct 18, 2024
8404787
Update docs/accessibility/get-started/introduction.mdx
marktnoonan Oct 18, 2024
f6e27f7
Update docs/app/guides/accessibility-testing.mdx
marktnoonan Oct 18, 2024
0cdd5f4
Update docs/app/guides/accessibility-testing.mdx
marktnoonan Oct 18, 2024
36225b4
Apply suggestions from code review
marktnoonan Oct 18, 2024
6621c14
cleanup and clarity
marktnoonan Oct 18, 2024
2106ca4
Apply suggestions from code review
marktnoonan Oct 18, 2024
32585c7
add managing performance section
marktnoonan Oct 18, 2024
2ded988
Merge branch 'a11y-ga-updates' of https://github.com/cypress-io/cypre…
marktnoonan Oct 18, 2024
43ddb22
add how it works section
marktnoonan Oct 19, 2024
df4dbf9
update text and images
marktnoonan Oct 19, 2024
d25fc6e
add test replay link
marktnoonan Oct 19, 2024
cb55822
mention ARIA, other minor edits
marktnoonan Oct 19, 2024
88e7923
tweaks
marktnoonan Oct 19, 2024
4f4369c
remove summary that isn't needed
marktnoonan Oct 19, 2024
e16af31
Merge branch 'ga-ui-cov-a11y' into a11y-ga-updates
marktnoonan Oct 19, 2024
9189618
lint fix
marktnoonan Oct 19, 2024
94a25eb
fix bad link
marktnoonan Oct 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions docs/accessibility/configuration/axe-core-configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,23 @@

Configuration for Axe-Core® rules is available through your Account Executive. We we are happy to have a call with you to dial in your report config to make sure you are getting the most useful reports possible, and we find this onboarding very effective.

Generally configuration of these rules isn't needed because your implementation of any policies about what should "fail a build" is handled using the [results-api.md](/accessibility/results-api), where you have full control over how to parse the results and what rules need to be reacted to. Keeping the results in Cypress Cloud broad helps you to still be able to see and understand all of the accessibility information, even if only a subset of the results would be considered blocking.
In most cases, configuration of these rules as they run in Cypress Cloud isn't needed, because your implementation of any policies about what should "fail a build" is handled using the [Accessibility Results API](/accessibility/results-api), where you have full control over how to parse the results and what rules need to be reacted to. Keeping the results in Cypress Cloud broad helps you to still be able to see and understand all of the accessibility information, even if only a subset of the results would be considered blocking.

**Note.** Certain Axe Core rules are off by default. Here are the rules that are currently either ignored by configuration, or for which results cannot be detected.
**Note.** Certain Axe Core® rules are off by default. Here are the rules that are currently either ignored by configuration, or for which results cannot be detected.

- `Elements must meet minimum color contrast ratio thresholds`
- This is off by default. This is both the slowest rule in the Axe Core ruleset, and the one that can have the highest percentage of false positives or incomplete checks.
- This is off by default. This is both the slowest rule in the Axe Core® ruleset, and the one that can have the highest percentage of false positives or incomplete checks.
- It does work perfectly in many projects, so we are happy to turn this on for you if requested, so that you can see the results.
- `<video> or <audio> elements must not play automatically`
- Ignored because Test Replay does not yet capture the metadata to check this
- `Delayed refresh under 20 hours must not be used`
- Ignored because Test Replay does not collect the metadata for this

### Component Testing
## Component Testing

Cypress Accessibility works great with component testing. In addition the the rules that are off by default, "page-level" rules do not run for component tests, since a component is usually a fragment mounted in an empty page that would not be expected to have things like headings or a `main` element.

## Updates to the Axe Core® library

Updates to the Axe Core® library are released several times each year. While Cypress does not commit to updating our internal Axe Core® version within a specific timeframe, we will always wait at least 30 days before making an update.
Since some updates introduce new rules or address bugs that change how issues are detected, this buffer is intended to give you time to make changes in your use of our [Results API](/accessibility/results-api) if needed. Axe Core® releases can be tracked [in GitHub](https://github.com/dequelabs/axe-core/releases).
4 changes: 2 additions & 2 deletions docs/accessibility/core-concepts/accessibility-score.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ sidebar_position: 30

Cypress produces a top-line percentage score that indicates a general sense of accessibility for a project. This overall score for a run is based on the **average score of** **all application snapshots** that were captured during the run, and is rooted in the ratio of passing and failing accessibility checks performed on each snapshot, weighted according to the severity of the issues.&#x20;

The severity levels presented in Cypress Cloud match the Axe Core `impact` level for each rule. This impact classification was developed by the creators of Axe, Deque Systems, to aid in prioritization of accessibility results, by highlighting rules that are likely to reveal the most substantial barriers (such as missing label text) so they can be addressed first.
The severity levels presented in Cypress Cloud match the Axe Core® `impact` level for each rule. This impact classification was developed by the creators of Axe, Deque Systems, to aid in prioritization of accessibility results, by highlighting rules that are likely to reveal the most substantial barriers (such as missing label text) so they can be addressed first.

It's important to note that the Web Content Accessibility Guidelines (WCAG) standards do not have a concept of relative severity of the Success Criteria. For **compliance** purposes, all failures of the Success Criteria count and must be addressed. For **usability** purposes, however, the impact values provided by Axe Core provide excellent guidance about which issues to take up first to bring the most benefit to disabled users of your plaform as quickly as possible.
It's important to note that the Web Content Accessibility Guidelines (WCAG) standards do not have a concept of relative severity of the Success Criteria. For **compliance** purposes, all failures of the Success Criteria count and must be addressed. For **usability** purposes, however, the impact values provided by Axe Core® provide excellent guidance about which issues to take up first to bring the most benefit to disabled users of your plaform as quickly as possible.

## Algorithm

Expand Down
34 changes: 34 additions & 0 deletions docs/accessibility/core-concepts/how-it-works.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
sidebar_label: 'How it works'
title: 'How it works'
description: Cypress Accessibility\'s checks are powered by Axe Core® from Deque Systems.
sidebar_position: 40
---

# How it works

The stages of the Cypress Accessibility process are as follows:

1. As your run is recorded and specs are uploaded to Cypress Cloud, accessibility checks begin, based on the incoming [Test Replay](/cloud/features/test-replay) data, on the Cypress Cloud servers.
1. When your run completes, the final specs are processed for accessibility checks. All earlier specs have already entered processing and are likely completed by the end of the run.
1. Cypress Cloud organizes all the the Views (pages and components) that were rendered during the run, merging and deduplicating across tests so that page-level results and live DOM snapshots can be grouped together.
1. A combined "rule-level" report is created.
1. The completed results are displayed in Cypress Cloud and are flagged in integrations such Slack as GitHub/Gitlab comments, and available CI via the Results API so that you can optionally fail your pipeline if your standards are not met.

## About Axe Core®

Axe Core® is an open source library for detecting accessibility violations. It is used widely in the industry and have over 1 billion total downloads on npm.
The library may be familiar from other tools, such as Lighthouse, or tools used for code linting, mobile testing and more, and has also been used extensively in Cypress tests through open-source packages (discussed more our [accessibility testing guide](/app/guides/accessibility-testing)). This means that Cypress Accessibility's
configuration can often be easily aligned with your existing accessibility standards across different teams and platforms.

Axe Core® checks for accessibility violations in your website's user interface, based on common accessibility standards. A violation detected by Axe usually indicates that some users with disabilities would have a problem using or perceiving a feature. An example of a violation would be an image without alternative text provided in the markup, which means the content is not perceivable to visually-impaired users. [Learn more about Axe.](https://github.com/dequelabs/axe-core)

While this process reaches a high standard for automated testing, it cannot test every possible aspect of compliance, especially for heavily custom interfaces or aspects requiring human judgment.

This means that a 100% axe score does not mean all possible accessibility errors have been ruled out. Deque reports that [Axe Core® can detect up to 57% of the issues that would appear in a manual accessibility audit](https://www.deque.com/blog/automated-testing-study-identifies-57-percent-of-digital-accessibility-issues/).

The value of this form of testing in Cypress Accessibility is to give you fast, reliable, easy-to-understand feedback about common accessibility mistakes that are found in most projects. Providing these results automatically as part of your test run means that you can find and fix these issues with minimal friction, shifting accessibility left in your software development lifecycle.

## Test Replay limitations

Because Cypress Accessibility uses data captured through the Test Replay protocol, it is subject to any [limitations of that protocol](https://docs.cypress.io/faq/questions/cloud-faq#Is-everything-captured-and-replayed-in-Test-Replay) related to data capture or browser support.
10 changes: 5 additions & 5 deletions docs/accessibility/core-concepts/run-level-reports.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,20 @@ An overall score is calculated based on all the Snapshots for a given View. The
The count of elements in a view that failed a Rule, had an inconclusive check for a Rule, or were explicitly ignored by configuration.

- **Failed** elements had at least one accessibility Rule fail during the run. These are considered in your accessibility Score and must be fixed.&#x20;
- **Inconclusive** elements had at least one check marked "incomplete" by Axe Core during the run. These represent checks that couldn't be completed for technical reasons, or items that need manual review to confirm if there is an accessibility issue. These do not affect your score.
- **Inconclusive** elements had at least one check marked "incomplete" by Axe Core® during the run. These represent checks that couldn't be completed for technical reasons, or items that need manual review to confirm if there is an accessibility issue. These do not affect your score.
- **Ignored** elements are those that had either failing or incomplete checks, but were ignored with [CSS selectors in your configuration](/accessibility/configuration/elementfilters). They do not affect your score. Cypress still processes these elements and makes the results available, so that you always know results were was ignored, since these elements are still present in your UI and users will encounter them.

#### Failed rules

Failed rules are broken down for each View with a mini chart displaying Axe Core severity levels, as well as a total count for failed rules. Both of these help you identify at a glance which pages or components have high numbers serious issues.
Failed rules are broken down for each View with a mini chart displaying Axe Core® severity levels, as well as a total count for failed rules. Both of these help you identify at a glance which pages or components have high numbers serious issues.

#### Severity

In Cypress Accessibility, severity levels are mapped to Axe Core's `impact` level for each rule. It's important to note that these levels were created by Deque Systems, the creators of Axe Core, and not a part of the Web Content Accessibility Guidelines themselves. For more details on this, see our [accessibility score documentation](https://github.com/dequelabs/axe-core/blob/develop/doc/issue_impact.md).

## Rules

This section lists the outcome of every Axe Core rule that was enabled at the time of this run.
This section lists the outcome of every Axe Core® rule that was enabled at the time of this run.

<DocsImage
src="/img/accessibility/core-concepts/cypress-accessibility-rules-list.png"
Expand All @@ -81,11 +81,11 @@ Each Rule in the table has the following attributes:
- An icon representing the overall status for the rule, which can be any one of these:
- **Passed** -- every time this rule applied to an element, the element either passed that rule's checks or was intentionally ignored
- **Failed** -- at least one failing element was detected in the run
- **Inconclusive** -- no elements failed this rule, but at least one element was labelled "incomplete" by Axe Core and requires manual review
- **Inconclusive** -- no elements failed this rule, but at least one element was labelled "incomplete" by Axe Core® and requires manual review
- **Inapplicable** -- this rule was enabled during the run, but there were no elements detected that required the rule to run
- **Ignored by config** -- this rule was ignored at the config level for the project and did not run
- The description for the rule
- The Axe Core severity level for the rule
- The Axe Core® severity level for the rule
- The counts of failed, inconclusive, and ignored elements for each rule

## Filtering
Expand Down
30 changes: 19 additions & 11 deletions docs/accessibility/get-started/introduction.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
sidebar_label: Introduction
title: 'Introduction'
description: 'Cypress Accessibility provides organized, visual, and actionable accessibility reports, based completely on the tests you record to Cypress Cloud, and powered by Axe Core by Deque Systems.'
description: 'Cypress Accessibility provides organized, visual, and actionable accessibility reports, based completely on the tests you record to Cypress Cloud, and powered by Axe Core® by Deque Systems.'
sidebar_position: 10
sidebar_custom_props: { 'new_label': true }
---
Expand All @@ -10,27 +10,35 @@ sidebar_custom_props: { 'new_label': true }

<AccessibilityAddon />

Cypress Accessibility provides organized, visual, and actionable accessibility reports, based completely on the tests you record to Cypress Cloud, and powered by Axe Core by Deque Systems.
## What is Cypress Accessibility?

Cypress Accessibility provides organized, visual, and actionable accessibility reports, based completely on the tests you record to Cypress Cloud, and powered by Axe Core® by Deque Systems. This is the documentation page for that product. To learn about the general topic of accessibility testing using the Cypress App, see [our guide on this topic](/app/guides/accessibility-testing).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻


Cypress Accessibility generates a sortable, filterable report, to see scores and violation counts by page/component, or a combined rule-based report for the entire run. It also includes a drill-in view with live DOM snapshots highlighting each issue detected.

<DocsImage
src="/img/accessibility/get-started/cypress-accessibility-overview.png"
alt="A graphic showing the Cypress Accessibility dashboard including list of rules, views, and the user interface showing how devtools can be used"
/>

## How it works
## How are reports created?

Cypress Cloud generates an accessibility report for each unique state reached during your tests on a given Cypress run. This includes pages visited in end-to-end tests, as well as individual components rendered in component tests, and any states or pages that were encountered as a result of interactions performed during test execution.

These individual accessibility "snapshots" are grouped into page- or component-level reports (called "Views"), and also form the bases of a unified report covering the entire run. This gives you multiple ways to examine your report - by focusing on the most important parts of your application, or by
All of these page or component states are detected using the same protocol that powers [Test Replay](/cloud/features/test-replay), meaning no additional code or setup is needed anywhere in your tests.

Reports are generated using the industry-standard open source [Axe Core](https://github.com/dequelabs/axe-core)® library created by Deque Systems, with a default configuration focused on the [Web Content Accessibility Guidelines (WCAG) 2.1](https://www.w3.org/WAI/standards-guidelines/wcag/).&#x20;

## Understanding what is tested
These individual accessibility "snapshots" are grouped into page- or component-level reports (called "Views"), and also form the bases of a unified report covering the entire run. This gives you multiple ways to examine your report - by focusing on the most important parts of your application, or by

Axe Core checks for accessibility violations in your website's user interface, based on common accessibility standards. A violation detected by Axe usually indicates that some users with disabilities would have a problem using or perceiving a feature. An example of a violation would be an image without alternative text provided in the markup, which means the content is not perceivable to visually-impaired users. [Learn more about Axe.](https://github.com/dequelabs/axe-core)
Reports are generated using the industry-standard open source [Axe Core®](https://github.com/dequelabs/axe-core) library created by Deque Systems, with a default configuration focused on the [Web Content Accessibility Guidelines (WCAG) 2.1](https://www.w3.org/WAI/standards-guidelines/wcag/).

Axe reaches a high standard for automated testing, but cannot test every possible aspect of compliance, especially for heavily custom interfaces or aspects requiring human judgment. Please note that a 100% axe score does not mean all possible accessibility errors have been ruled out. [Axe typically detects up to 57% of the issues that would appear in a manual accessibility audit.](https://www.deque.com/blog/automated-testing-study-identifies-57-percent-of-digital-accessibility-issues/)
## Highlights

##### Trademark notice
- All steps of all user flows tested in Cypress are captured automatically
- Server-side execution means no disruption to existing test code or practices, and no performance impact from accessibility checks
- Reports are generated at the page or component level, as well as a combined report showing the outcome of all accessibility rules checked in the run
- Debuggable full-page DOM snapshots with highlights are provided for every violation
- Most iframe and Shadow DOM uses cases are supported out-of-the-box
- The [Results API](/accessibility/results-api) allows you to own how your CI pipeline reacts to the accessibility results of the run to implement a policy
- "Ignored" elements are isolated and don't affect your score, but can still be reviewed

_Note: DEQUE, DEQUELABS, AXE®, and AXE-CORE® are trademarks of Deque Systems, Inc. in the US and other countries._
_Note: DEQUE, DEQUELABS, AXE®, and AXE-CORE® are trademarks of Deque Systems, Inc. in the US and other countries. These terms are used throughout our documentation to refer to the accessibility testing library used by Cypress Accessibility._
12 changes: 10 additions & 2 deletions docs/app/core-concepts/best-practices.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -220,16 +220,24 @@ If the answer is **no** because the text could be changed - then use

Cypress loves the Testing Library project. We use Testing Library internally,
and our philosophy aligns closely with Testing Library's ethos and approach to
writing tests. We strongly endorse their best practices.
writing tests. We strongly endorse their best practices for situations where,
as with `cy.contains()`, you want to fail a test if a specific piece of content or
accessible role is not present.

You can use the
[Cypress Testing Library](https://testing-library.com/docs/cypress-testing-library/intro/)
package to use the familiar testing library methods (like `findByRole`,
`findByLabelText`, etc...) to select elements in Cypress specs.

In particular, if you're looking for more resources to understand how we
If you are coming from a React Testing Library background and looking for more resources to understand how we
recommend you approach testing your components, look to: [Cypress Component Testing](/app/component-testing/get-started).

### Accessibility Testing

Selecting elements with data attributes, text content, or Testing Library locators can each have some different implications for accessibility,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

but none of these approaches is a "complete" accessibility test, and you will always need additional, accessibility-specific testing (including automated and manual tests) to confirm
your application is working as expected for people with disabilities and the technology they use. See[our accessibility testing guide](/app/guides/accessibility-testing) for more details and comparisons of approaches.

## <Icon name="angle-right" /> Assigning Return Values

:::danger
Expand Down
Loading