-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Add Cypress Accessibility Branch Review documentation #6110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
bf8e0c4
fix link to axe-core changelog
marktnoonan 90c7eba
add accessibility branch review notes
marktnoonan 08aa1d3
add images, update changelog & score docs
marktnoonan 07e9099
rename section and add video
marktnoonan 4e51899
fix broken link
marktnoonan fe434a5
Merge branch 'main' into marktnoonan/accessibility-branch-review
marktnoonan 2e8cd1a
lint fix
marktnoonan 4e0667a
update how it works
marktnoonan 4423b83
minor updates
marktnoonan 562e674
Update docs/accessibility/core-concepts/comparing-reports.mdx
marktnoonan 43d2075
Update docs/accessibility/core-concepts/comparing-reports.mdx
marktnoonan 6c0b28f
Update docs/accessibility/core-concepts/comparing-reports.mdx
marktnoonan 85ea691
Update docs/accessibility/guides/improve-accessibility.mdx
marktnoonan 88d12a2
text fixes
marktnoonan 9feb2f3
Update docs/accessibility/core-concepts/comparing-reports.mdx
marktnoonan a16aa13
Merge branch 'main' into marktnoonan/accessibility-branch-review
marktnoonan a047220
lint fix
marktnoonan 3944dd6
Merge branch 'marktnoonan/accessibility-branch-review' of https://git…
marktnoonan fed4f3e
Merge branch 'main' into marktnoonan/accessibility-branch-review
marktnoonan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,108 @@ | ||
--- | ||
title: 'Comparing Accessibility Reports | Cypress Documentation' | ||
description: 'Review the main areas to pay attention to when first reviewing an accessibility report for a Cypress run.' | ||
sidebar_position: 40 | ||
sidebar_label: Compare reports | ||
--- | ||
|
||
<ProductHeading product="accessibility" /> | ||
|
||
# Compare reports | ||
|
||
Accessibility reports from different runs can be compared in the [Branch Review](/cloud/features/branch-review#Getting-Started) area of Cypress Cloud. This allows you to instantly see if any new issues have been introduced, and drill in to see full-page HTML and CSS snapshots showing only the new issues. | ||
|
||
See the video below for an example of using Branch Review in a Pull Request: | ||
|
||
<DocsVideo | ||
src="https://www.youtube.com/embed/k3vq57O4P4M" | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
title="Accessibility branch review demo" | ||
/> | ||
|
||
## Use cases | ||
|
||
Comparing the results from different runs is useful in multiple scenarios. | ||
|
||
**Key use cases:** | ||
|
||
- **Pre-merge checks**: Know if any net-new issues are introduced by UI code changes. | ||
- **Monitoring changes**: Compare nightly monitoring runs and track down the introduction of new problems caused by underlying changes in the application. | ||
- **Detecting content issues**: Sometimes content editors can introduce accessibility issues unrelated to code changes. Seeing the example issues presented visually, in context, helps you quickly triage whether you are dealing with a recent code change issue, or a content authorship problem. | ||
- **Reviewing AI-generated code changes**: The increased use of AI to generate and/or review front-end code creates some increased risks of accessibility regressions making it to production. The increase or decrease of accessibility issues when reviewing a pull request helps you understand the impact of the change. | ||
- **Tracing the introduction of issues**: With dropdowns for each run, it's easy to rapidly compare different A and B runs to find the exact commit that introduced a problem. | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Demonstrating the resolution of issues**: Confirm the effect of your improvements, and share overview with your team to more quickly review code changes. | ||
|
||
## Content of the report | ||
|
||
<DocsImage | ||
src="/img/accessibility/core-concepts/branch-review-intro.png" | ||
alt="Cypress Accessibility Branch Review with arrows calling out the sections for new failed rules, resolved rules, and failed rules with changes. New failed rules are defined as rules that just started failing on the new branch. Resolved rules are defined as rules that got fully fixed in the new branch. Failed rules with changes are defined as rules that failed on both branches, but with different element counts or different related pages." | ||
noBorder={true} | ||
/> | ||
|
||
The Branch Review report is prioritized into three sections: | ||
|
||
- **New failed rules**: Rules that were passing every time they ran on the `base` run but now have failures in the `changed` run. | ||
- **Resolved rules**: Rules that had some failures detected in the `base` run but have no failures on the `changed` run. This helps you celebrate the wins and recognize when a new run has gone green. | ||
- **Failed rules with changes**: Rules that were failing in both runs, where the elements with failures detected have increased, decreased, or changed in some way. | ||
|
||
This organization of the report brings the most significant results to the top - regressions of rules that had been fully passing in the `base` run. Increases or decreases in the element counts for rules that already have some failures provide a good sense of progress, but can tend to be noisier, especially if the runs have different content or other conditions have changed in between. | ||
|
||
## How to compare runs | ||
|
||
The first step is to get to the Branch Review area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed. We refer to be baseline fun for comparison as the `base` run, and the changes we are comparing with as the `changed` run. | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
There are a number of ways to get to Branch Review depending on where you are looking at your Cypress results. In all cases, once you've picked a `changed` run, you can adjust the automatically-selected `base` run to any branch or run in the project. | ||
|
||
### From a Pull Request | ||
|
||
<DocsImage | ||
src="/img/accessibility/core-concepts/branch-review-pull-request.png" | ||
alt="A pull request comment from Cypress with an arrow pointing to the 'View all changes introduced in this branch' link. Test results and some accessibility score and violation information is also visible in the comment. The project has an 86.86% accessibility score, with 9 failed elements. Failed rule counts are 1 critical, 1 serious, 3 moderate, and 0 minor." | ||
/> | ||
|
||
Click the "View all changes introduced in this branch" link at the bottom of the Test Results table. You will enter branch review with the current PR's branch and latest run pre-selected as the "changed" run. | ||
|
||
### From the run list | ||
|
||
Click the branch name associated with the run. This will take you to Branch Review with that branch's newest run pre-selected as the `changed` run. | ||
|
||
<DocsImage | ||
src="/img/accessibility/core-concepts/branch-review-run-card.png" | ||
alt="A couple of cards showing various details about Cypress test runs, including pass/fail counts and metadata. An arrow points to the branch name." | ||
/> | ||
|
||
### From the project list | ||
|
||
<DocsImage | ||
src="/img/accessibility/core-concepts/branch-review-project-card.png" | ||
width="40%" | ||
alt="A card showing the name of an example project and related branches to choose." | ||
/> | ||
|
||
The project card shows three active branches for the project. You can click through to any of these to enter Branch Review with the newest run on that branch pre-selected as the `changed` run. | ||
|
||
### From the main navigation | ||
|
||
<DocsImage | ||
src="/img/accessibility/core-concepts/branch-review-side-nav.png" | ||
width="40%" | ||
alt="The main Cypress Cloud navigation showing Latest Runs, Branches, and Analytics menu items. An arrow points to Branches." | ||
/> | ||
|
||
When inside of a project, you can select "Branches" in the main navigation to see a full, filterable list of available branches and choose one to set as the `changed` run. | ||
|
||
## FAQ | ||
|
||
### How do I ensure a good comparison? | ||
|
||
The best subjects to compare are passing runs that ran similar tests on the same set of content. This means that each run visited roughly the same pages and completed the same kinds of workflows. In this situation, any diff in the results is likely the result of changes present in the newer run. This is usually what happens out-of-the box when comparing a pull-request branch with your main branch. | ||
|
||
That said, it still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results. | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### What is the purpose of the Beta label? | ||
|
||
This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please hit the feedback button and let us know. | ||
|
||
### Why do I see some views (pages or components) changing from run-to-run? | ||
|
||
URLs with dynamic slugs in them can appear as "new" pages in some situations. This behavior can be adjusted with [View configuration](/accessibility/configuration/views) to make sure page names will match across runs by wildcarding parts of the URL as needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
docs/accessibility/core-concepts/inspecting-violation-details.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+48.5 KB
static/img/accessibility/core-concepts/branch-review-project-card.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.