[SYCL][E2E] Track amount of XFAILed tests
#15603
Conversation
We have a problem with our approach to `XFAIL`ing failed tests - we don't always submit trackers for those failures, thus simply hiding them from us. Some of tests we `XFAIL`ed have been in that state for years and without a tracker submitted about them, we don't even know that there are issues. This patch introduces a new requirement for marking test as expected to fail: every `XFAIL` directive has to be followed by `XFAIL-TRACKER` which points to intel/llvm issue submitted to analyze the failure and remove `XFAIL` directive. The tracker can be referenced either by URL, or through GH shortcuts `owner/repo#NNNNN`. To ensure that the new requirement is followed, a new test was added which checks amount of improper `XFAIL` directives (i.e. those which are not followed by `XFAIL-TRACKER` directive). Similar approach is expected to be applied later for `UNSUPPORTED` directives, but it will be done as a separate PR.
AlexeySachkov
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
AlexeySachkov
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
…k-amount-of-xfailed-tests-without-tracker
AlexeySachkov
had a problem deploying
to
WindowsCILock
GitHub Actions
Error
Co-authored-by: Marcos Maronas <[email protected]>
AlexeySachkov
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
AlexeySachkov
had a problem deploying
to
WindowsCILock
GitHub Actions
Error
sycl/test-e2e/README.md
Outdated
| ``` | ||
| // GPU driver update caused failure | ||
| // XFAIL: level_zero | ||
| // XFAIL-TRACKER: intel/llvm#DDDDD |
There was a problem hiding this comment.
Just wondering - can't we proceed with the full link instead? Looks like I need to go to the "issues", open some and then paste the number to find the ticket (is there another way?)
upd. or I can use the search line. Anyway, It would be more convenient to have a link
There was a problem hiding this comment.
The format allows full links here and, it is just that example uses a shortcut. I agree that it will be easier to have a full link, so let me make it stricter
…tracker' of https://github.com/AlexeySachkov/llvm into private/asachkov/track-amount-of-xfailed-tests-without-tracker
AlexeySachkov
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
AlexeySachkov
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
…k-amount-of-xfailed-tests-without-tracker
AlexeySachkov
had a problem deploying
to
WindowsCILock
GitHub Actions
Error
|
Local merge highlighted that amount of |
|
Pre-commit passed, I just did a local merge merge with |
AlexeySachkov
deleted the
private/asachkov/track-amount-of-xfailed-tests-without-tracker
branch
5 participants
We have a problem with our approach to
XFAILing failed tests - we don't always submit trackers for those failures, thus simply hiding them from us.Some of tests we
XFAILed have been in that state for years and without a tracker submitted about them, we don't even know that there are issues.This patch introduces a new requirement for marking test as expected to fail: every
XFAILdirective has to be followed byXFAIL-TRACKERwhich points to intel/llvm issue submitted to analyze the failure and removeXFAILdirective.The tracker can be referenced either by URL, or through GH shortcuts
owner/repo#NNNNN.To ensure that the new requirement is followed, a new test was added which checks amount of improper
XFAILdirectives (i.e. those which are not followed byXFAIL-TRACKERdirective).Similar approach is expected to be applied later for
UNSUPPORTEDdirectives, but it will be done as a separate PR.