Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/accessibility/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@ sidebar_position: 200

# Changelog

## Week of 2/24/2025

- [Cypress Accessibility Branch Review](/accessibility/core-concepts/comparing-runs) has entered beta. Read the docs to learn how to compare runs and automatically spot newly introduced issues.

## Week of 1/13/2025

- Cypress Accessibility results are now included in the Enterprise Reporting area of Cypress Cloud. Filter by project and branch and see trends over time.

## Week of 11/11/2024

- A "Share Issue" button has been added to the element details section for each reported rule violation. This allows for easily copying the key details and related links for a particular issue into your clipboard to add to a message or to a ticket in your issue tracking system.
Expand Down
6 changes: 4 additions & 2 deletions docs/accessibility/core-concepts/accessibility-score.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: 'Accessibility score | Cypress Documentation'
description: 'Cypress produces a top-line percentage score that indicates a general sense of accessibility for a project.'
sidebar_label: Accessibility score
sidebar_position: 30
sidebar_position: 40
---

<ProductHeading product="accessibility" />
Expand All @@ -19,7 +19,7 @@ _Note: WCAG standards do not categorize Success Criteria by severity; all failur

## Scoring algorithm

The accessibility score calculation applies the following weights:
The accessibility score calculation applies the following weights to failed accessibility checks in a snapshot.

| Severity of Issue | Weight |
| ----------------- | ------ |
Expand All @@ -36,6 +36,8 @@ The accessibility score calculation applies the following weights:
- Calculate the total weight of failed checks.
- Compute the score as: `Score = (Passed Weights) / (Passed Weights + Failed Weights)`

**Note**: Incomplete and Ignored element statuses do not have any effect on your score. Only clear failures of a rule are counted.

2. **View Score:**

- Average the scores of all snapshots within the View.
Expand Down
110 changes: 110 additions & 0 deletions docs/accessibility/core-concepts/comparing-runs.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
---
title: 'Accessibility Run-level Reports | Cypress Documentation'
description: 'Review the main areas to pay attention to when first reviewing an accessibility report for a Cypress run.'
sidebar_position: 40
sidebar_label: Comparing runs
---

<ProductHeading product="accessibility" />

# Comparing runs

Accessibility reports from different runs can be compared in the [Branch Review](/cloud/features/branch-review#Getting-Started) area of Cypress Cloud. This allows you to instantly see if any new issues have been introduced, and drill in to see full-page HTML and CSS snapshots showing only the new issues.

<DocsImage
src="/img/accessibility/core-concepts/branch-review-example-diff.png"
alt="An example of a Cypress Accessibility report comparison in the Branch Review area. The example shows no new failed rules, no resolved rules, and one failed rule with changes - the 'form elements must have labels' rule, which has two new failures on the `/contact` route."
/>


## Use cases

Comparing runs can be useful in various scenarios.

**Key use cases:**

- **Pre-merge checks**: Know if any net-new issues are introduce by UI code changes.
- **Monitoring changes**: Compare nightly monitoring runs and track down the introduction of new problems caused by underlying changes in the application.
- **Detecting content issues**: Sometimes content editors can introduce accessibility issues unrelated to code changes. Seeing the example issues presented visually, in context, helps you quickly triage whether you are dealing with a recent code change issue, or a content authorship problem.
- **Reviewing AI-generated code changes**: The increased use of AI to generate and/or review front-end code creates some increased risks of accessibility regressions making it to production. Seeing the increase or decrease of accessibility issues when reviewing a pull request.


## Content of the report

<DocsImage
src="/img/accessibility/core-concepts/branch-review-intro.png"
alt="Cypress Accessibility Branch Review with arrows calling out the sections for new failed rules, resolved rules, and failed rules with changes. New failed rules are defined as rules that just started failing on the new branch. Resolved rules are defined as rules that got fully fixed in the new branch. Failed rules with changes are defined as rules that failed on both branches, but with different element counts or different related pages."
/>


The Branch Review report is prioritized into three sections:

- **New failed rules**: Rules that were passing every time they ran on the `base` run but now have failures in the `changed` run.
- **Resolved rules**: Rules that had some failures detected in the `base` run but have no failures on the `changed` run. This helps you celebrate the wins and recognize when a new run has gone green.
- **Failed rules with changes**: Rules that were failing in both runs, where the elements with failures detected have increased, decreased, or changed in some way.

This organization of the report brings the most significant results to the top - regressions of rules that had been fully passing in the `base` run. Increases or decreased in the element counts for rules that already have some failures provide a good sense of progress, but can tend to be noisier, especially if the runs have different content or other conditions that changed in between.

## How to compare runs

The first step is to get to the Branch Review area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed. We refer to be baseline fun for comparison as the `base` run, and the changes we are comparing with as the `changed` run.

There are a number of ways to get to Branch Review depending on where you are looking at your Cypress results. In all cases, once you've picked a `changed` run, you can adjust the automatically-selected `base` run to any branch or run in the project.

### From a Pull Request

<DocsImage
src="/img/accessibility/core-concepts/branch-review-pull-request.png"
alt="A pull request comment from Cypress with an arrow pointing to the 'View all changes introduced in this branch' link. Test results and some accessibility score and violation information is also visible in the comment. The project has an 86.86% accessibility score, with 9 failed elements. Failed rule counts are 1 critical, 1 serious, 3 moderate, and 0 minor."
/>


Click the "View all changes introduced in this branch" link at the bottom of the Test Results table. You will enter branch review with the current PR's branch and latest run pre-selected as the "changed" run.

### From the run list

Click the branch name associated with the run. This will take you to Branch Review with that branch's newest run pre-selected as the `changed` run.

<DocsImage
src="/img/accessibility/core-concepts/branch-review-run-card.png"
alt="A couple of cards showing various details about Cypress test runs, including pass/fail counts and metadata. An arrow points to the branch name."
/>


### From the project list

<DocsImage
src="/img/accessibility/core-concepts/branch-review-project-card.png"
width="40%"
alt="A card showing the name of an example project and related branches to choose."
/>


The project card shows three active branches for the project. You can click through to any of these to enter Branch Review with the newest run on that branch pre-selected as the `changed` run.

### From the main navigation

<DocsImage
src="/img/accessibility/core-concepts/branch-review-side-nav.png"
width="40%"
alt="The main Cypress Cloud navigation showing Latest Runs, Branches, and Analytics menu items. An arrow points to Branches."
/>


When inside of a project, you can select "Branches" in the main navigation to see a full, filterable list of available branches and choose one to set as the `changed` run.

## FAQ

### How to I ensure a good comparison?
The more similar the runs being compared are in terms of content that was rendered during the test runs, the more precise the diff will be. For this reason, it's ideal to compare passing runs that were fairly close to each other in time. If either run being compared has failing tests, this can introduce noise because certain pages or elements might not have been rendered in both runs and will appear as new or missing.

That said, it still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results.

### What is the purpose of the Beta label?
This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please hit the feedback button and let us know.

### Why do I see some views (pages or components) changing from run-to-run?
URLs with dynamic slugs in them can appear as "new" pages in some situations. This behavior can be adjusted with View configuration.

### Can we fail runs automatically based on new violations?
While it is possible to establish a baseline for comparison in the Results API, this is not wired up directly to the branch review.
4 changes: 2 additions & 2 deletions docs/accessibility/core-concepts/how-it-works.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: 'Cypress Accessibility | How it works'
description: 'Accessibility checks are powered by Axe Core® from Deque Systems based on common accessibility standards.'
sidebar_label: 'How it works'
sidebar_position: 40
sidebar_position: 50
---

<ProductHeading product="accessibility" />
Expand Down Expand Up @@ -33,7 +33,7 @@ The value of this form of testing in Cypress Accessibility is to give you fast,

Axe Core® is regularly updated, with Cypress following a controlled adoption process:

- Update Policy: Cypress Cloud integrates new Axe Core® versions at least 30 days after their release. This provides teams time to review the Cypress Accessibility [changelog](https://github.com/dequelabs/axe-core/blob/develop/CHANGELOG.md) so you can make any necessary adjustments to your process in advance of Cypress Cloud bumping the version.
- Update Policy: Cypress Cloud integrates new Axe Core® versions at least 30 days after their release. This provides teams time to review the Axe Core® [changelog](https://github.com/dequelabs/axe-core/blob/develop/CHANGELOG.md) so you can make any necessary adjustments to your process in advance of Cypress Cloud bumping the version.
- Changelog Tracking: Updates are documented in the Cypress Accessibility [changelog](https://github.com/dequelabs/axe-core/blob/develop/CHANGELOG.md), starting December 1, 2024.
- Critical Exceptions: In cases like critical security patches, updates may occur sooner, with notifications provided to affected customers.

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.