configure number of passes before a flake is marked exempt?

[jluhrsen]

It looks like flaking cases will be marked exempt after they have
passed 5 times in a row:

https://help.testspace.com/dashboard/space-failures#exemptions

Is it possible to configure this? I know my team would like to
keep track over a much longer time frame (months, hundreds of
CI runs) to try to catch and fix our flakes.

[munderseth]

Hi @jluhrsen, at this time you can not configure how many consecutive passes transitions a test case out of the flaky state.

The way the system works now, once a test case is tagged as flaky, it requires 5 consecutive passes to transition out. For example, if it passes 4 times in a row and then fails, it is still tagged as flaky and the requirement for 5 consecutive passes is reset.

An interesting graph, that might be helpful, is Test Failures - https://help.testspace.com/dashboard/space-metrics#test-failures. You can review, over time, the CI's Flaky rate.

[jluhrsen]

Thank you for the reply @munderseth. Do you think this would be something that ever would land as a
configurable knob?

in our use case we have flakes that happen but not very frequently. They are still painful to the team as
they don't know if their work (PR's in this case) are hitting a known flake or some new problem their PR
has introduced. tracking and resolving the flakes is becoming important for us.

here is my fork just running the same code a couple times a day and you can see flakes coming and
going.

Ideally what we are looking for is over a long period of time to know which cases have the highest
flake rate so we can know where to focus and prioritize the debugging and fixing.

Oh, and yes I have seen our failure tracks graph. it is helpful to see that visually.