Skip to content

add "flaky tests" summary to Github Actions Reporter #9498

@dominikg

Description

@dominikg

Clear and concise description of the problem

Sometimes tests are flaky and people use the retry option to avoid failing CI when that happens.
But you still want to know which tests are flaky to try and improve them.

Suggested solution

Collect flaky tests (reruns > 0) in the reporter and print a warning summary at the end of the run.

The reason for suggesting the warning summary at the end is that this is obviously less important than failures and shouldn't be intermittent with other errors or warnings in the output.

Ideally link to the test by line and if possible also link back to the execution in the log (no idea if the latter is possible)

Alternative

A custom reporter could be built to provide this

Additional context

discord discussion https://discord.com/channels/917386801235247114/1462789015895019724

Validations

Metadata

Metadata

Assignees

Projects

Status

Approved

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions