More specific exit codes for "cypress run" and log flaky tests in run result summary #26166
Description
What would you like?
According to https://docs.cypress.io/guides/guides/command-line#Exit-code there are only 2 types of exit codes:
- "0": no failures
- "> 0": amount of tests that failed
In Gitlab CI it's possible to let a job pass with warnings using "allow_failure" together with "exit_codes" ( https://docs.gitlab.com/ee/ci/yaml/#allow_failureexit_codes ).
What I'm missing is a specific exit code for test flake. I wish to see static exit codes such as:
- 0: no failures
- 1: has test failures (with or without flaky tests)
- 2: no failures, but flaky tests
Furthermore flaky tests should be logged separately in the run result summary
Why is this needed?
It would be possible to make the developer aware that test run had flaky tests and that he should investigate even though overall the tests passed. This way he would know if he introduced a flaky test or detected a new flaky test.
Also having flaky tests listed in the summary is much easier than to browsing the artifacts.
When using sorry-cypress oder Cypress Dashboard/Cloud this is generally less of an issue because you could get the information from their APIs respectively. But not everybody uses sorry-cypress/Cypress Cloud and I think all users would benefit from this feature.
Also it is important that Cypress collects the information for the flaky tests. Otherwise every persons own implementation could possibly break after an update.
Other
No response