Add report-only mode for accessibility violations #30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces the ability to collect and report accessibility violations
without failing tests. This is useful for existing applications with
many violations where you want to audit without breaking the test suite.
Configuration:
* `audit_enabled = :stdout` - Log violations to stdout
* `audit_enabled = { file: 'path' }` - Export to JSON file with
structured violation data including severity, WCAG tags, affected
elements, and remediation guidance
The JSON report groups violations by page and by rule, with a summary
showing total violations and pages affected. Reports are output once at
the end of the test run via `Minitest.after_run` or RSpec's
`after(:suite)` hook.
Maintains full backwards compatibility:
* `audit_enabled = true` - Assert mode (default, existing behavior)
* `audit_enabled = false` - Disabled (existing behavior)
Implementation details:
* New `ReportMode` class with subclasses for each mode (Disabled,
Assert, StdoutReporter, FileReporter)
* New `Reporter` class collects violations and generates reports
* `assert_no_accessibility_violations` always asserts (even when
auto-audits are disabled or in report mode)
* Automatic audits respect the configured report mode
* Includes comprehensive test coverage with golden file test
gasi
changed the title
seanpdoyle
added a commit
that referenced
this pull request
Dec 19, 2025
Related to #21 Related to #22 Related to #30 Related to #30 Rather than immediately fail tests (like the `:raise` reporter), log the output through a new `:log` reporter. --- > How many I conduct a preliminary audit that comprehensively exercises > my system test suite without failing tests that result in violations You can configure the audit's reporting mechanism. By default, the `config.capybara_accessibility_audit.reporter` value is set to `:raise`, which will raise violation errors that will fail the test suite. To log violations, rather than raise them, you can configure `config.capybara_accessibility_audit.reporter` to `:log`: ```ruby class MySystemTest < ApplicationSystemTestCase self.accessibility_audit_reporter = :log test "with overridden accessibility :log reporter" do visit examples_path # ... end end ``` --- `report.capybara_accessibility_audit` notification --- Subscribe to `report.capybara_accessibility_audit` notifications emitted when an accessibility audit detects violations. The `payload` includes: | Payload | Type | Description | | ------------- | ---------------------------------- | ----------- | | report | [Axe::API::Results][] | The underlying axe.js [Results][] object | options | [ActiveSupport::OrderedOptions][] | The audit's configuration | test | [ActionDispatch::SystemTestCase][] | The test case that triggered the audit [ActiveSupport::OrderedOptions]: https://api.rubyonrails.org/classes/ActiveSupport/OrderedOptions.html [ActionDispatch::SystemTestCase]: https://api.rubyonrails.org/classes/ActionDispatch/SystemTestCase.html [Axe::API::Results]: https://github.com/dequelabs/axe-core-gems/blob/v4.11.0/packages/axe-core-api/lib/axe/api/results.rb [Results]: https://www.deque.com/axe/core-documentation/api-documentation/#results-object > [!NOTE] > The `report.capybara_accessibility_audit` notifications are only published when > `config.capybara_accessibility_audit.reporter` is configured with > `:notification` or `:log`.
seanpdoyle
added a commit
that referenced
this pull request
Dec 19, 2025
Related to #21 Related to #22 Related to #30 Related to #30 Rather than immediately fail tests (like the `:raise` reporter), log the output through a new `:log` reporter. --- > How many I conduct a preliminary audit that comprehensively exercises > my system test suite without failing tests that result in violations You can configure the audit's reporting mechanism. By default, the `config.capybara_accessibility_audit.reporter` value is set to `:raise`, which will raise violation errors that will fail the test suite. To log violations, rather than raise them, you can configure `config.capybara_accessibility_audit.reporter` to `:log`: ```ruby class MySystemTest < ApplicationSystemTestCase self.accessibility_audit_reporter = :log test "with overridden accessibility :log reporter" do visit examples_path # ... end end ``` --- `report.capybara_accessibility_audit` notification --- Subscribe to `report.capybara_accessibility_audit` notifications emitted when an accessibility audit detects violations. The `payload` includes: | Payload | Type | Description | | ------------- | ---------------------------------- | ----------- | | report | [Axe::API::Results][] | The underlying axe.js [Results][] object | options | [ActiveSupport::OrderedOptions][] | The audit's configuration | test | [ActionDispatch::SystemTestCase][] | The test case that triggered the audit [ActiveSupport::OrderedOptions]: https://api.rubyonrails.org/classes/ActiveSupport/OrderedOptions.html [ActionDispatch::SystemTestCase]: https://api.rubyonrails.org/classes/ActionDispatch/SystemTestCase.html [Axe::API::Results]: https://github.com/dequelabs/axe-core-gems/blob/v4.11.0/packages/axe-core-api/lib/axe/api/results.rb [Results]: https://www.deque.com/axe/core-documentation/api-documentation/#results-object > [!NOTE] > The `report.capybara_accessibility_audit` notifications are only published when > `config.capybara_accessibility_audit.reporter` is configured with > `:notification` or `:log`.
seanpdoyle
added a commit
that referenced
this pull request
Dec 19, 2025
Related to #21 Related to #22 Related to #30 Related to #30 Rather than immediately fail tests (like the `:raise` reporter), log the output through a new `:log` reporter. --- > How many I conduct a preliminary audit that comprehensively exercises > my system test suite without failing tests that result in violations You can configure the audit's reporting mechanism. By default, the `config.capybara_accessibility_audit.reporter` value is set to `:raise`, which will raise violation errors that will fail the test suite. To log violations, rather than raise them, you can configure `config.capybara_accessibility_audit.reporter` to `:log`: ```ruby class MySystemTest < ApplicationSystemTestCase self.accessibility_audit_reporter = :log test "with overridden accessibility :log reporter" do visit examples_path # ... end end ``` --- `report.capybara_accessibility_audit` notification --- Subscribe to `report.capybara_accessibility_audit` notifications emitted when an accessibility audit detects violations. The `payload` includes: | Payload | Type | Description | | ------------- | ---------------------------------- | ----------- | | report | [Axe::API::Results][] | The underlying axe.js [Results][] object | options | [ActiveSupport::OrderedOptions][] | The audit's configuration | test | [ActionDispatch::SystemTestCase][] | The test case that triggered the audit [ActiveSupport::OrderedOptions]: https://api.rubyonrails.org/classes/ActiveSupport/OrderedOptions.html [ActionDispatch::SystemTestCase]: https://api.rubyonrails.org/classes/ActionDispatch/SystemTestCase.html [Axe::API::Results]: https://github.com/dequelabs/axe-core-gems/blob/v4.11.0/packages/axe-core-api/lib/axe/api/results.rb [Results]: https://www.deque.com/axe/core-documentation/api-documentation/#results-object > [!NOTE] > The `report.capybara_accessibility_audit` notifications are only published when > `config.capybara_accessibility_audit.reporter` is configured with > `:notification` or `:log`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the ability to collect and report accessibility violations without failing tests, making it easier to adopt accessibility auditing in existing applications with many violations.
Configuration options
:stdout- Log violations to stdout{ file: 'path' }- Export structured JSON reports with full violation details including severity, WCAG tags, affected elements, and remediation guidancetrue/falsevaluesKey benefits
JSON export features