Cache tests fail after Linux GPU reset

[sarnex]

Describe the bug

If we start doing a GPU reset before starting E2E tests on Linux BMG, these tests fail for some reason

Failed Tests (3):
  SYCL :: KernelAndProgram/persistent-cache-multi-device.cpp
  SYCL :: ProgramManager/multi_device_bundle/build_twice.cpp
  SYCL :: ProgramManager/multi_device_bundle/device_libs_and_caching.cpp

FAIL: SYCL :: ProgramManager/multi_device_bundle/device_libs_and_caching.cpp (530 of 2285)
******************** TEST 'SYCL :: ProgramManager/multi_device_bundle/device_libs_and_caching.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 17: env NEOReadDebugKeys=1 CreateMultipleRootDevices=4 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/device_libs_and_caching.cpp.tmp_jit.out | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp --check-prefixes=CHECK-SPIRV-JIT-LINK-TRACE
+ env NEOReadDebugKeys=1 CreateMultipleRootDevices=4 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/device_libs_and_caching.cpp.tmp_jit.out
+ /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp --check-prefixes=CHECK-SPIRV-JIT-LINK-TRACE
UR ---> platform->initialize()
UR <--- platform->initialize()(UR_RESULT_SUCCESS)
UR ---> Device->initialize()
UR <--- Device->initialize()(UR_RESULT_SUCCESS)
/__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp:68:33: error: CHECK-SPIRV-JIT-LINK-TRACE: expected string not found in input
 // CHECK-SPIRV-JIT-LINK-TRACE: urProgramCreateWithIL(
                                ^
<stdin>:1:1: note: scanning from here
 ---> urAdapterGet
^
<stdin>:1:7: note: possible intended match here
 ---> urAdapterGet
      ^

Input file: <stdin>
Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/device_libs_and_caching.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1:  ---> urAdapterGet 
check:68'0     X~~~~~~~~~~~~~~~~~~ error: no match found
check:68'1           ?             possible intended match
            2:  <--- urAdapterGet(.NumEntries = 0, .phAdapters = nullptr, .pNumAdapters = 0x7ffe6c5ee704 (1)) -> UR_RESULT_SUCCESS; 
check:68'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            3:  ---> urAdapterGet 
check:68'0     ~~~~~~~~~~~~~~~~~~~
            4:  <--- urAdapterGet(.NumEntries = 1, .phAdapters = 0x8037f30 {0x8036670}, .pNumAdapters = nullptr) -> UR_RESULT_SUCCESS; 
check:68'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            5:  ---> urAdapterGetInfo 
check:68'0     ~~~~~~~~~~~~~~~~~~~~~~~
            6:  <--- urAdapterGetInfo(.hAdapter = 0x8036670, .propName = UR_ADAPTER_INFO_BACKEND, .propSize = 4, .pPropValue = 0x7ffe6c5ee6f8 (UR_ADAPTER_BACKEND_LEVEL_ZERO), .pPropSizeRet = nullptr) -> UR_RESULT_SUCCESS; 
check:68'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            .
            .
            .
>>>>>>

--

********************
FAIL: SYCL :: KernelAndProgram/persistent-cache-multi-device.cpp (1246 of 2285)
******************** TEST 'SYCL :: KernelAndProgram/persistent-cache-multi-device.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 5: env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_CACHE_PERSISTENT=1 SYCL_CACHE_TRACE=1 SYCL_CACHE_DIR=/__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp/cache_dir env -u XDG_CACHE_HOME env -u HOME env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp.out 2>&1 | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp --check-prefixes=CHECK-BUILD
+ env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_CACHE_PERSISTENT=1 SYCL_CACHE_TRACE=1 SYCL_CACHE_DIR=/__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp/cache_dir env -u XDG_CACHE_HOME env -u HOME env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/KernelAndProgram/Output/persistent-cache-multi-device.cpp.tmp.out
+ /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp --check-prefixes=CHECK-BUILD
/__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:30:18: error: CHECK-BUILD: expected string not found in input
 // CHECK-BUILD: [Persistent Cache]: device binary has been cached
                 ^
<stdin>:1:1: note: scanning from here
persistent-cache-multi-device.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:25: int main(): Assertion `devs.size() >= 3' failed.
^
<stdin>:1:94: note: possible intended match here
persistent-cache-multi-device.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:25: int main(): Assertion `devs.size() >= 3' failed.
                                                                                             ^

Input file: <stdin>
Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: persistent-cache-multi-device.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/KernelAndProgram/persistent-cache-multi-device.cpp:25: int main(): Assertion `devs.size() >= 3' failed. 
check:30'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:30'1                                                                                                  ?                                                                                        possible intended match
>>>>>>

--

********************
FAIL: SYCL :: ProgramManager/multi_device_bundle/build_twice.cpp (1397 of 2285)
******************** TEST 'SYCL :: ProgramManager/multi_device_bundle/build_twice.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/build_twice.cpp.tmp.out | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp
+ env NEOReadDebugKeys=1 CreateMultipleRootDevices=3 SYCL_UR_TRACE=2 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/ProgramManager/multi_device_bundle/Output/build_twice.cpp.tmp.out
+ /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp
UR ---> platform->initialize()
UR <--- platform->initialize()(UR_RESULT_SUCCESS)
UR ---> Device->initialize()
UR <--- Device->initialize()(UR_RESULT_SUCCESS)
/__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp:32:12: error: CHECK: expected string not found in input
 // CHECK: urProgramCreateWithIL(
           ^
<stdin>:1:1: note: scanning from here
 ---> urAdapterGet
^
<stdin>:1:7: note: possible intended match here
 ---> urAdapterGet
      ^

Input file: <stdin>
Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/ProgramManager/multi_device_bundle/build_twice.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1:  ---> urAdapterGet 
check:32'0     X~~~~~~~~~~~~~~~~~~ error: no match found
check:32'1           ?             possible intended match
            2:  <--- urAdapterGet(.NumEntries = 0, .phAdapters = nullptr, .pNumAdapters = 0x7ffe0a7e85b4 (1)) -> UR_RESULT_SUCCESS; 
check:32'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            3:  ---> urAdapterGet 
check:32'0     ~~~~~~~~~~~~~~~~~~~
            4:  <--- urAdapterGet(.NumEntries = 1, .phAdapters = 0x200561a0 {0x200548e0}, .pNumAdapters = nullptr) -> UR_RESULT_SUCCESS; 
check:32'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            5:  ---> urAdapterGetInfo 
check:32'0     ~~~~~~~~~~~~~~~~~~~~~~~
            6:  <--- urAdapterGetInfo(.hAdapter = 0x200548e0, .propName = UR_ADAPTER_INFO_BACKEND, .propSize = 4, .pPropValue = 0x7ffe0a7e85a8 (UR_ADAPTER_BACKEND_LEVEL_ZERO), .pPropSizeRet = nullptr) -> UR_RESULT_SUCCESS; 
check:32'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            .
            .
            .
>>>>>>

--

To reproduce

1. Include a code snippet that is as short as possible
2. Specify the command which should be used to compile the program
3. Specify the command which should be used to launch the program
4. Indicate what is wrong and what was expected

Environment

• OS: [e.g Windows/Linux]
• Target device and vendor: [e.g. Intel GPU]
• DPC++ version: [e.g. commit hash or output of clang++ --version]
• Dependencies version: [e.g. the output of sycl-ls --verbose]

Additional context

No response