sycl-ls errors with SYCL Exception

[sycloid]

Describe the bug

1. Download nightly bundle: wget https://github.com/intel/llvm/releases/download/nightly-2025-09-07/sycl_linux.tar.gz
2. Unpack it to a prefix
3. Add <prefix>/bin to PATH and <prefix>/lib to LD_LIBRARY_PATH

Now execute sycl-ls runs into errors:

$ sycl-ls --verbose
<LOADER>[INFO]: loaded adapter 0x0x560d45ab1a80 (libur_adapter_cuda.so.0) from /home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_cuda.so.0
<LOADER>[INFO]: failed to load adapter 'libur_adapter_hip.so.0' with error: libamd_comgr.so.2: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter '/home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_hip.so.0' with error: libamd_comgr.so.2: cannot open shared object file: No such file or directory
<LOADER>[INFO]: loaded adapter 0x0x560d45ab7700 (libur_adapter_level_zero.so.0) from /home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_level_zero.so.0
<LOADER>[INFO]: loaded adapter 0x0x560d45ab6b00 (libur_adapter_level_zero_v2.so.0) from /home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_level_zero_v2.so.0
<LOADER>[INFO]: failed to load adapter 'libur_adapter_native_cpu.so.0' with error: libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter '/home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_native_cpu.so.0' with error: /home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter 'libur_adapter_offload.so.0' with error: libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter '/home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_offload.so.0' with error: /home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: loaded adapter 0x0x560d45aba220 (libur_adapter_opencl.so.0) from /home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_opencl.so.0
[FATAL UMF] utils_open_library: dlopen(libcuda.so) failed with error: libcuda.so: cannot open shared object file: No such file or directory
[FATAL UMF] umfCUDAMemoryProviderParamsCreate: Loading CUDA symbols failed
<CUDA>[ERROR]:
UR ERROR:
        Value:           UR_RESULT_ERROR_UNKNOWN
        Function:        operator()
        Source Location: /__w/llvm/llvm/src/unified-runtime/source/adapters/cuda/platform.cpp:98

SYCL Exception encountered: cuda backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)

Note that excluding CUDA device, sycl-ls works as expected:

$ ONEAPI_DEVICE_SELECTOR="opencl:*;level_zero:*" sycl-ls
INFO: Output filtered by ONEAPI_DEVICE_SELECTOR environment variable, which is set to opencl:*;level_zero:*.
To see device ids, use the --ignore-device-selectors CLI option.

[level_zero:gpu] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Graphics [0x7d45] 12.70.4 [1.6.31294.120000]
[opencl:gpu] Intel(R) OpenCL Graphics, Intel(R) Graphics [0x7d45] OpenCL 3.0 NEO  [25.27.34303.5]

To reproduce

1. Include code snippet as short as possible
2. Specify the command which should be used to compile the program
3. Specify the command which should be used to launch the program
4. Indicate what is wrong and what was expected

Environment

• OS: Ubunutu 24.04 on WSL
• Target device and vendor: [e.g. Nvidia GPU]
• DPC++ version: [e.g. commit hash or output of clang++ --version]

$ clang++ --version
clang version 21.0.0git (https://github.com/intel/llvm 25fbd1f710d9f5b1426b447e45ba8a7b07ab739b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/sycloid/sycl-builds/07-09-2025/bin
Build config: +assertions

• Dependencies version: [e.g. the output of nvidia-smi and sycl-ls --verbose]

$ nvidia-smi
Sun Sep  7 16:40:26 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.02              Driver Version: 581.15         CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050 ...    On  |   00000000:02:00.0 Off |                  N/A |
| N/A   43C    P0             11W /   35W |       0MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Additional context

I experience the same problem with oneAPI DPC++ 2025.2

[bratpiorka]

Hi @sycloid, thank you for submitting this bug.
Does the problem still occur if you add libcuda to LD_LIBRARY_PATH?

[sycloid]

@bratpiorka Yes, it does.

$ whereis libcuda.so
libcuda.so: /usr/lib/wsl/lib/libcuda.so
$ LD_LIBRARY_PATH=/usr/lib/wsl/lib/libcuda.so:$LD_LIBRARY_PATH sycl-ls
[FATAL UMF] utils_open_library: dlopen(libcuda.so) failed with error: libcuda.so: cannot open shared object file: No such file or directory
[FATAL UMF] umfCUDAMemoryProviderParamsCreate: Loading CUDA symbols failed
<CUDA>[ERROR]:
UR ERROR:
        Value:           UR_RESULT_ERROR_UNKNOWN
        Function:        operator()
        Source Location: /__w/llvm/llvm/src/unified-runtime/source/adapters/cuda/platform.cpp:98

SYCL Exception encountered: cuda backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN) 

[sycloid]

Actually, my bad. Adding the folder to LD_LIBRARY_PATH, not the library path itself resolves the issue:

$ LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH sycl-ls
[cuda:gpu][cuda:0] NVIDIA CUDA BACKEND, NVIDIA GeForce RTX 3050 4GB Laptop GPU 8.6 [CUDA 13.0]
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Graphics [0x7d45] 12.70.4 [1.6.31294.120000]
[opencl:cpu][opencl:0] Intel(R) OpenCL, Intel(R) Core(TM) Ultra 7 155U OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Graphics [0x7d45] OpenCL 3.0 NEO  [25.27.34303.5]

Curiously, using LD_PRELOAD does not work:

$ LD_PRELOAD=/usr/lib/wsl/lib/libcuda.so sycl-ls
[FATAL UMF] utils_open_library: dlopen(libcuda.so) failed with error: libcuda.so: cannot open shared object file: No such file or directory
[FATAL UMF] umfCUDAMemoryProviderParamsCreate: Loading CUDA symbols failed
<CUDA>[ERROR]:
UR ERROR:
        Value:           UR_RESULT_ERROR_UNKNOWN
        Function:        operator()
        Source Location: /__w/llvm/llvm/src/unified-runtime/source/adapters/cuda/platform.cpp:98

SYCL Exception encountered: cuda backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)

[bratpiorka]

I made a quick fix #20071 - @sycloid could you please test it?
It is not merged yet, so you need to build llvm from sources.

git clone -b rrudnick_syclls_umf_cuda_load_fix https://github.com/bratpiorka/llvm.git --depth=1 ~/test_llvm
cd ~/test_llvm
python3 buildbot/configure.py -t Release -o build_sycl --cuda
cmake --build build_sycl -j 100

export LD_LIBRARY_PATH=~/test_llvm/build_sycl/lib:$LD_LIBRARY_PATH 
export PATH=~/test_llvm/build_sycl/bin:$PATH

sycl-ls

[sycloid]

I had built the branch rrudnick_syclls_umf_cuda_load_fix.

The sycl-ls command now works as expected:

$ sycl-ls
[cuda:gpu][cuda:0] NVIDIA CUDA BACKEND, NVIDIA GeForce RTX 3050 4GB Laptop GPU 8.6 [CUDA 13.0]
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Graphics [0x7d45] 12.70.4 [1.6.31294.120000]
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Graphics [0x7d45] OpenCL 3.0 NEO  [25.27.34303.5]

$ clang++ --version
clang version 21.0.0git (https://github.com/bratpiorka/llvm.git 46e7d83aee2fef78a4b02d048c14232286171851)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/sycloid/sycl-builds/test_llvm/build_sycl/bin
Build config: +assertions

[bratpiorka]

great - I'll continue working on merging the fix and let you know when it's ready

[bratpiorka]

hi @sycloid - the fix has been merged.
Could you please check the latest nightly build?

[bratpiorka]

closing the issue as the problem no longer occurs