Repeat tests until it passes once, temporary solution to flaky tests #4575
Unanswered
s-dakshin
asked this question in
Community Help
Replies: 1 comment
-
I'd welcome such feature also |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Goal: I am trying to solve flakiness in UI tests in our custom test framework using Google Test. Such that I can re-run tests until they pass at least once. Like have them repeat 3 times and if they pass even once, mark the test as passed. The opposite of this is possible with the
::testing::GTEST_FLAG(repeat) = 3;
What I am trying to achieve is based on a blog post - https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html. Where the suggestions to flaky tests are -
Based on this, it seems like what I am trying to achieve should be possible. But based on the things I tried, here are my findings -
testing::FLAGS_gtest_repeat
- seems to be used to run tests until it fails to help with debugging. I almost want the inverse of this.testing::TestPartResult
- seems to have a way to find if fatal or non-fatal errors occurred in your test. But there is no way to clear these flags or handle it inside our code. More precisely,::testing::Test::HasFailure()
is set to failure even if a test fails once, I want something liketesting::TestResult::ClearTestPartResults()
, but that is a private function.Can someone please help/suggest a way to achieve my goal using existing functions/flags that Google Test framework offers?
Beta Was this translation helpful? Give feedback.
All reactions