[SYCL][E2E] Track amount of XFAILed tests

sycl/test-e2e/README.md

@@ -320,3 +320,27 @@ implementation header files is still in progress and the final set of these
 "fine-grained" includes that might be officially documented and suggested for
 customers to use isn't determined yet. **Until then, code outside of this project
 must keep using `<sycl/sycl.hpp>` provided by the SYCL2020 specification.**
+
+## Marking tests as expected to fail
+
+Every test should be written in a way that it is either passed, or it is skipped
+(in case it is not compatible with an environment it was launched in).
+
+If for any reason you find yourself in need to temporary mark test as expected
+to fail under certain conditions, you need to submit an issue to the repo to
+analyze that failure and make test passed or skipped.
+
+Once the issue is created, you can update the test by adding `XFAIL` and
+`XFAIL-TRACKER` directive:
+```
+// GPU driver update caused failure
+// XFAIL: level_zero
+// XFAIL-TRACKER: PRJ-5324
+
+// Sporadically fails on CPU:
+// XFAIL: cpu
+// XFAIL-TRACKER: https://github.com/intel/llvm/issues/DDDDD
+```
+
+If you add `XFAIL` without `XFAIL-TRACKER` directive,
+`no-xfail-without-tracker.cpp` test will fail, notifying you about that.

sycl/test-e2e/Sampler/normalized-clampedge-linear-float.cpp

@@ -3,6 +3,7 @@
 // RUN: %{build} -o %t.out
 // RUN: %{run} %t.out
 // XFAIL: hip
+// XFAIL-TRACKER: https://github.com/intel/llvm/issues/14732
 
 // CUDA works with image_channel_type::fp32, but not with any 8-bit per channel
 // type (such as unorm_int8)

sycl/test-e2e/Sampler/normalized-clampedge-nearest.cpp

@@ -5,6 +5,7 @@
 // Missing __spirv_ImageWrite, __spirv_SampledImage,
 // __spirv_ImageSampleExplicitLod on AMD
 // XFAIL: hip_amd
+// XFAIL-TRACKER: https://github.com/intel/llvm/issues/14732
 
 /*
     This file sets up an image, initializes it with data,

sycl/test-e2e/no-xfail-without-tracker.cpp

@@ -0,0 +1,50 @@
+// This test is intended to ensure that we have no trackers marked as XFAIL
+// without a tracker information added to a test.
+//
+// The format we check is:
+// XFAIL: lit,features
+// XFAIL-TRACKER: [GitHub issue URL|Internal tracker ID]
+//
+// GitHub issue URL format:
+//     https://github.com/owner/repo/issues/12345
+//
+// Internal tracker ID format:
+//     PROJECT-123456
+//
+// REQUIRES: linux
+//
+// Explanation of the command:
+// - search for all "XFAIL" occurrences, display line with match and the next one
+//   -I, --include to drop binary files and other unrelated files
+// - in the result, search for "XFAIL" again, but invert the result - this
+//   allows us to get the line *after* XFAIL
+// - in those lines, check that XFAIL-TRACKER is present and correct. Once
+//   again, invert the search to get all "bad" lines
+// - make a final count of how many ill-formatted directives there are and
+//   verify that against the reference
+//
+// RUN: grep -rI "XFAIL:" %S -A 1 --include=*.c --include=*.cpp \
+// RUN:     --no-group-separator | \
+// RUN: grep -v "XFAIL:" | \
+// RUN: grep -Pv "XFAIL-TRACKER:\s+(?:https://github.com/[\w\d-]+/[\w\d-]+/issues/[\d]+)|(?:[\w]+-[\d]+)" | \
+// RUN: wc -l | FileCheck %s --check-prefix NUMBER-OF-XFAIL-WITHOUT-TRACKER
+//
+// The number below is a number of tests which are *improperly* XFAIL-ed, i.e.
+// we either don't have a tracker associated with a failure listed in those
+// tests, or it is listed in a wrong format.
+// Note: strictly speaking, that is not amount of files, but amount of XFAIL
+// directives. If a test contains several XFAIL directives, some of them may be
+// valid and other may not.
+//
+// That number *must not* increase. Any PR which causes this number to grow
+// should be rejected and it should be updated to either keep the number as-is
+// or have it reduced (preferrably, down to zero).
+//
+// If you see this test failed for your patch, it means that you either
+// introduced XFAIL directive to a test improperly, or broke the format of an
+// existing XFAIL-ed tests.
+// Another possibility (and that is a good option) is that you updated some
+// tests to match the required format and in that case you should just update
+// (i.e. reduce) the number below.
+//
+// NUMBER-OF-XFAIL-WITHOUT-TRACKER: 187