Allow marking tests Pending when they fail audits #22
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Works on thoughtbot#21. This doesn't completely fix the problem that is identified in thoughtbot#21, but it is a step in that direction. The idea is that, if the configuration setting `accessibility_audit_skip_on_error` is set to true, if an `axe` violation is detected, instead of generating an error it will instead mark the test as `skipped` with the `axe` output as the message in the skip output. There is a problem with this approach, which is the same problem identified in issue thoughtbot#7 - as soon as a single test is marked skipped or failed, then the rest of the test is skipped. This means that if a test has both failing application logic and failing accessibility validation, the first of those tests to fail will be the one that is reported. For a system test, since any interaction with the page will trigger the accessibility tests, it is likely that the test will be marked as skipped and the application logic failure will be hidden. This can be solved by running the tests a second time with `accessibility_audit_enabled = false` and comparing the results, but that does require two runs through the test suite.
Contributor
|
Thank you so much for opening #21 along with this PR. We really appreciate your contributions. At the moment, I think the difficulty you're experiencing is signaling a larger need. I think the project would benefit from the concept of a "reporter" for controlling whether or not (or when) to raise failures for violations. I'm imagining something like RSpec's configuration to control "failing fast". I'd characterize the current behavior as failing fast. To work toward deferred failures, I'm imagining a "reporter" with at least two hooks: 1) an "on violation" hook and 2) an "on completion" hook. A backwards compatible reporter could flunk the tests immediately in its "on violation" hook, then do nothing in its "on completion" hook. A reporter that behaves like you've described might collect the violation into an internal data structure (maybe as simple as an A third reporter might not fail the test "on completion", but could instead output a report to a file, or do something else entirely. I've pushed up a violation-reporter branch to explore where the seams might be to insert a reporter. It isn't merge-ready, but might provide some insight. |
seanpdoyle
added a commit
that referenced
this pull request
Dec 19, 2025
Related to #21 Related to #22 Related to #30 Related to #30 Rather than immediately fail tests (like the `:raise` reporter), log the output through a new `:log` reporter. --- > How many I conduct a preliminary audit that comprehensively exercises > my system test suite without failing tests that result in violations You can configure the audit's reporting mechanism. By default, the `config.capybara_accessibility_audit.reporter` value is set to `:raise`, which will raise violation errors that will fail the test suite. To log violations, rather than raise them, you can configure `config.capybara_accessibility_audit.reporter` to `:log`: ```ruby class MySystemTest < ApplicationSystemTestCase self.accessibility_audit_reporter = :log test "with overridden accessibility :log reporter" do visit examples_path # ... end end ``` --- `report.capybara_accessibility_audit` notification --- Subscribe to `report.capybara_accessibility_audit` notifications emitted when an accessibility audit detects violations. The `payload` includes: | Payload | Type | Description | | ------------- | ---------------------------------- | ----------- | | report | [Axe::API::Results][] | The underlying axe.js [Results][] object | options | [ActiveSupport::OrderedOptions][] | The audit's configuration | test | [ActionDispatch::SystemTestCase][] | The test case that triggered the audit [ActiveSupport::OrderedOptions]: https://api.rubyonrails.org/classes/ActiveSupport/OrderedOptions.html [ActionDispatch::SystemTestCase]: https://api.rubyonrails.org/classes/ActionDispatch/SystemTestCase.html [Axe::API::Results]: https://github.com/dequelabs/axe-core-gems/blob/v4.11.0/packages/axe-core-api/lib/axe/api/results.rb [Results]: https://www.deque.com/axe/core-documentation/api-documentation/#results-object > [!NOTE] > The `report.capybara_accessibility_audit` notifications are only published when > `config.capybara_accessibility_audit.reporter` is configured with > `:notification` or `:log`.
seanpdoyle
added a commit
that referenced
this pull request
Dec 19, 2025
Related to #21 Related to #22 Related to #30 Related to #30 Rather than immediately fail tests (like the `:raise` reporter), log the output through a new `:log` reporter. --- > How many I conduct a preliminary audit that comprehensively exercises > my system test suite without failing tests that result in violations You can configure the audit's reporting mechanism. By default, the `config.capybara_accessibility_audit.reporter` value is set to `:raise`, which will raise violation errors that will fail the test suite. To log violations, rather than raise them, you can configure `config.capybara_accessibility_audit.reporter` to `:log`: ```ruby class MySystemTest < ApplicationSystemTestCase self.accessibility_audit_reporter = :log test "with overridden accessibility :log reporter" do visit examples_path # ... end end ``` --- `report.capybara_accessibility_audit` notification --- Subscribe to `report.capybara_accessibility_audit` notifications emitted when an accessibility audit detects violations. The `payload` includes: | Payload | Type | Description | | ------------- | ---------------------------------- | ----------- | | report | [Axe::API::Results][] | The underlying axe.js [Results][] object | options | [ActiveSupport::OrderedOptions][] | The audit's configuration | test | [ActionDispatch::SystemTestCase][] | The test case that triggered the audit [ActiveSupport::OrderedOptions]: https://api.rubyonrails.org/classes/ActiveSupport/OrderedOptions.html [ActionDispatch::SystemTestCase]: https://api.rubyonrails.org/classes/ActionDispatch/SystemTestCase.html [Axe::API::Results]: https://github.com/dequelabs/axe-core-gems/blob/v4.11.0/packages/axe-core-api/lib/axe/api/results.rb [Results]: https://www.deque.com/axe/core-documentation/api-documentation/#results-object > [!NOTE] > The `report.capybara_accessibility_audit` notifications are only published when > `config.capybara_accessibility_audit.reporter` is configured with > `:notification` or `:log`.
seanpdoyle
added a commit
that referenced
this pull request
Dec 19, 2025
Related to #21 Related to #22 Related to #30 Related to #30 Rather than immediately fail tests (like the `:raise` reporter), log the output through a new `:log` reporter. --- > How many I conduct a preliminary audit that comprehensively exercises > my system test suite without failing tests that result in violations You can configure the audit's reporting mechanism. By default, the `config.capybara_accessibility_audit.reporter` value is set to `:raise`, which will raise violation errors that will fail the test suite. To log violations, rather than raise them, you can configure `config.capybara_accessibility_audit.reporter` to `:log`: ```ruby class MySystemTest < ApplicationSystemTestCase self.accessibility_audit_reporter = :log test "with overridden accessibility :log reporter" do visit examples_path # ... end end ``` --- `report.capybara_accessibility_audit` notification --- Subscribe to `report.capybara_accessibility_audit` notifications emitted when an accessibility audit detects violations. The `payload` includes: | Payload | Type | Description | | ------------- | ---------------------------------- | ----------- | | report | [Axe::API::Results][] | The underlying axe.js [Results][] object | options | [ActiveSupport::OrderedOptions][] | The audit's configuration | test | [ActionDispatch::SystemTestCase][] | The test case that triggered the audit [ActiveSupport::OrderedOptions]: https://api.rubyonrails.org/classes/ActiveSupport/OrderedOptions.html [ActionDispatch::SystemTestCase]: https://api.rubyonrails.org/classes/ActionDispatch/SystemTestCase.html [Axe::API::Results]: https://github.com/dequelabs/axe-core-gems/blob/v4.11.0/packages/axe-core-api/lib/axe/api/results.rb [Results]: https://www.deque.com/axe/core-documentation/api-documentation/#results-object > [!NOTE] > The `report.capybara_accessibility_audit` notifications are only published when > `config.capybara_accessibility_audit.reporter` is configured with > `:notification` or `:log`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Works on #21. This is a draft, mostly to start the conversation for one way this could work.
This doesn't completely fix the problem that is identified in #21, but it is a step in that direction. The idea is that, if the configuration setting
accessibility_audit_skip_on_erroris set to true, if anaxeviolation is detected, instead of generating an error it will instead mark the test asskippedwith theaxeoutput as the message in the skip output.There is a problem with this approach, which is the same problem identified in issue #7 - as soon as a single test is marked skipped or failed, then the rest of the test is skipped. This means that if a test has both failing application logic and failing accessibility validation, the first of those tests to fail will be the one that is reported. For a system test, since any interaction with the page will trigger the accessibility tests, it is likely that the test will be marked as skipped and the application logic failure will be hidden.
This can be solved by running the tests a second time with
accessibility_audit_enabled = falseand comparing the results, but that does require two runs through the test suite.