[Offload] Unit tests fail with "Plugin was not initialized!" assertion in `deinit`

[leandrolcampos]

When I try to run the kernel.unittest test:

> ninja -C build/runtimes/runtimes-bins OffloadUnitTests
> ./build/runtimes/runtimes-bins/offload/unittests/OffloadAPI/kernel.unittests

I get this error message at the end of the test log:

kernel.unittests: /home/leandro/llvm-project/offload/plugins-nextgen/common/src/PluginInterface.cpp:1658: Error llvm::omp::target::plugin::GenericPluginTy::deinit(): Assertion `Initialized && "Plugin was not initialized!"' failed.
pure virtual method called
terminate called without an active exception
Aborted (core dumped)

The same happens for other tests, like device.unittests and memory.unittests.

Environment:

• Device: CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
• LLVM version: Custom build from git
      * LLVM Project Commit Hash: 2886d30dd68f798deda7093d00bf9fc52dabcdd7
• Operating System: Ubuntu 24.04.2 LTS (WSL2)

[leandrolcampos]

Hi, @callumfare and @RossBrunton.
Could you help me here, please?
All the best,

[llvmbot]

@llvm/issue-subscribers-offload

Author: Leandro Lacerda (leandrolcampos)

When I try to run the `kernel.unittest` test:

> ninja -C build/runtimes/runtimes-bins OffloadUnitTests
> ./build/runtimes/runtimes-bins/offload/unittests/OffloadAPI/kernel.unittests

I get this error message at the end of the test log:

kernel.unittests: /home/leandro/llvm-project/offload/plugins-nextgen/common/src/PluginInterface.cpp:1658: Error llvm::omp::target::plugin::GenericPluginTy::deinit(): Assertion `Initialized && "Plugin was not initialized!"' failed.
pure virtual method called
terminate called without an active exception
Aborted (core dumped)

The same happens for other tests, like device.unittests and memory.unittests.

Environment:

• Device: CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
• LLVM version: Custom build from git
      * LLVM Project Commit Hash: 2886d30dd68f798deda7093d00bf9fc52dabcdd7
• Operating System: Ubuntu 24.04.2 LTS (WSL2)

[RossBrunton]

I am unable to reproduce this on my AMD system, so this is likely an issue with the Nvidia plugin. @callumfare mind having a look?

I took a quick look and this jumped out at me:

Error GenericPluginTy::init() {
  if (Initialized)
    return Plugin::success();

  auto NumDevicesOrErr = initImpl();
  if (!NumDevicesOrErr)
    return NumDevicesOrErr.takeError();
  Initialized = true;

  [...]
}

If cuInit (or friend) encounters an error, should we really consider the plugin not initialised? I think it makes more sense for init to always unconditionally require a deinit.

[jhuber6]

Hm, I'm not sure what the expectations here are. I'd assume if it fails to initialize it's not counted as initialized since you can't use it. There's probably some documentation on that behavior in CUDA or something.

[callumfare]

Hi @leandrolcampos , I'm unable to reproduce this locally on the same commit. Could you try re-running the tests with tracing enabled?

OFFLOAD_TRACE=1 ./build/runtimes/runtimes-bins/offload/unittests/OffloadAPI/kernel.unittests

Additionally if you're able to run in a debugger and provide a backtrace that shows where the program aborts, that would be really helpful.

[callumfare]

I think one issue is that we currently don't check that olInit succeeds in the test environment (in OffloadInitWrapper)

[leandrolcampos]

@callumfare, see the test log when I enable tracing:

leandro@Zephyrus:~/llvm-project$ OFFLOAD_TRACE=1 ./build/runtimes/runtimes-bins/offload/unittests/OffloadAPI/kernel.unittests
()-> OL_SUCCESS
---> olIterateDevices---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734640 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc22734614 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olGetPlatformInfoSize(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_NAME, .PropSizeRet = 0x7ffc227345e0 (5))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_NAME, .PropSize = 5, .PropValue = 0x55f7f8b78790 (CUDA))-> OL_SUCCESS
---> olGetDeviceInfoSize(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_NAME, .PropSizeRet = 0x7ffc227345e0 (35))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_NAME, .PropSize = 35, .PropValue = 0x55f7f8b78690 (NVIDIA GeForce RTX 4070 Laptop GPU))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b746e0, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734640 (0x55f7f8331e10))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331e10, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc22734614 (OL_PLATFORM_BACKEND_HOST))-> OL_SUCCESS
(.Callback = 0x55f7c3b0d280, .UserData = 0x55f7c3be2840)-> OL_SUCCESS
[==========] Running 10 tests from 8 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from olLaunchKernelFooTest
[ RUN      ] olLaunchKernelFooTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olIterateDevices---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc227347e0 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227347dc (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b746e0, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc227347e0 (0x55f7f8331e10))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331e10, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227347dc (OL_PLATFORM_BACKEND_HOST))-> OL_SUCCESS
(.Callback = 0x55f7c3b0d510, .UserData = 0x55f7c3be2860)-> OL_SUCCESS
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8b79078 (0x55f7f8b90b00))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8b90c90, .ProgDataSize = 3444, .Program = 0x55f7f8b79088 (0x55f7f8b8e280))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8b8e280, .Name = 0x55f7c3baefa4 (foo), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8b790b8 (0x55f7f8c52ad0))-> OL_SUCCESS
---> olMemAlloc(.Device = 0x55f7f8b74650, .Type = OL_ALLOC_TYPE_MANAGED, .Size = 256, .AllocationOut = 0x7ffc22734838 (0x204c00000))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8b90b00, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c52ad0, .ArgumentsData = 0x7ffc22734828, .ArgumentsSize = 8, .LaunchSizeArgs = 0x55f7f8b79090 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8b90b00)-> OL_SUCCESS
---> olMemFree(.Address = 0x204c00000)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8b8e280)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8b90b00)-> OL_SUCCESS
[       OK ] olLaunchKernelFooTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (23 ms)
[ RUN      ] olLaunchKernelFooTest.SuccessSynchronous/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8c159d8 (0x55f7f8c14e20))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8c47b20, .ProgDataSize = 3444, .Program = 0x55f7f8c159e8 (0x55f7f8c60bf0))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8c60bf0, .Name = 0x55f7c3baefa4 (foo), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8c15a18 (0x55f7f8bb0f20))-> OL_SUCCESS
---> olMemAlloc(.Device = 0x55f7f8b74650, .Type = OL_ALLOC_TYPE_MANAGED, .Size = 256, .AllocationOut = 0x7ffc22734838 (0x204c00000))-> OL_SUCCESS
---> olLaunchKernel(.Queue = nullptr, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8bb0f20, .ArgumentsData = 0x7ffc22734828, .ArgumentsSize = 8, .LaunchSizeArgs = 0x55f7f8c159f0 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olMemFree(.Address = 0x204c00000)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8c60bf0)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8c14e20)-> OL_SUCCESS
[       OK ] olLaunchKernelFooTest.SuccessSynchronous/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (14 ms)
[----------] 2 tests from olLaunchKernelFooTest (37 ms total)

[----------] 1 test from olLaunchKernelNoArgsTest
[ RUN      ] olLaunchKernelNoArgsTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8c16aa8 (0x55f7f8339850))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8b98f40, .ProgDataSize = 2500, .Program = 0x55f7f8c16ab8 (0x55f7f8b96be0))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8b96be0, .Name = 0x55f7c3baf092 (noargs), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8c16ae8 (0x55f7f8c50e80))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8339850, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c50e80, .ArgumentsData = nullptr, .ArgumentsSize = 0, .LaunchSizeArgs = 0x55f7f8c16ac0 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8339850)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8b96be0)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8339850)-> OL_SUCCESS
[       OK ] olLaunchKernelNoArgsTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (9 ms)
[----------] 1 test from olLaunchKernelNoArgsTest (9 ms total)

[----------] 1 test from olLaunchKernelLocalMemTest
[ RUN      ] olLaunchKernelLocalMemTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8c16aa8 (0x55f7f8c55640))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8bb0fc0, .ProgDataSize = 5404, .Program = 0x55f7f8c16ab8 (0x55f7f8c17c40))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8c17c40, .Name = 0x55f7c3baf099 (localmem), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8c16ae8 (0x55f7f8c61450))-> OL_SUCCESS
---> olMemAlloc(.Device = 0x55f7f8b74650, .Type = OL_ALLOC_TYPE_MANAGED, .Size = 1024, .AllocationOut = 0x7ffc22734838 (0x204c00000))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8c55640, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c61450, .ArgumentsData = 0x7ffc22734830, .ArgumentsSize = 8, .LaunchSizeArgs = 0x55f7f8c16ac0 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 4, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 256}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8c55640)-> OL_SUCCESS
---> olMemFree(.Address = 0x204c00000)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8c17c40)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8c55640)-> OL_SUCCESS
[       OK ] olLaunchKernelLocalMemTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (18 ms)
[----------] 1 test from olLaunchKernelLocalMemTest (18 ms total)

[----------] 1 test from olLaunchKernelLocalMemReductionTest
[ RUN      ] olLaunchKernelLocalMemReductionTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8bd7c08 (0x55f7f8bd7c60))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8c4b030, .ProgDataSize = 5784, .Program = 0x55f7f8bd7c18 (0x55f7f8bbde30))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8bbde30, .Name = 0x55f7c3baf0a2 (localmem_reduction), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8bd7c48 (0x55f7f8c50c00))-> OL_SUCCESS
---> olMemAlloc(.Device = 0x55f7f8b74650, .Type = OL_ALLOC_TYPE_MANAGED, .Size = 16, .AllocationOut = 0x7ffc22734838 (0x204c00000))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8bd7c60, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c50c00, .ArgumentsData = 0x7ffc22734830, .ArgumentsSize = 8, .LaunchSizeArgs = 0x55f7f8bd7c20 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 4, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 256}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8bd7c60)-> OL_SUCCESS
---> olMemFree(.Address = 0x204c00000)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8bbde30)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8bd7c60)-> OL_SUCCESS
[       OK ] olLaunchKernelLocalMemReductionTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (19 ms)
[----------] 1 test from olLaunchKernelLocalMemReductionTest (19 ms total)

[----------] 1 test from olLaunchKernelLocalMemStaticTest
[ RUN      ] olLaunchKernelLocalMemStaticTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8c51bc8 (0x55f7f8c5bd90))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8c4b030, .ProgDataSize = 5776, .Program = 0x55f7f8c51bd8 (0x55f7f8bcf280))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8bcf280, .Name = 0x55f7c3baf0b5 (localmem_static), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8c51c08 (0x55f7f8c54950))-> OL_SUCCESS
---> olMemAlloc(.Device = 0x55f7f8b74650, .Type = OL_ALLOC_TYPE_MANAGED, .Size = 16, .AllocationOut = 0x7ffc22734838 (0x204c00000))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8c5bd90, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c54950, .ArgumentsData = 0x7ffc22734830, .ArgumentsSize = 8, .LaunchSizeArgs = 0x55f7f8c51be0 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 4, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8c5bd90)-> OL_SUCCESS
---> olMemFree(.Address = 0x204c00000)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8bcf280)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8c5bd90)-> OL_SUCCESS
[       OK ] olLaunchKernelLocalMemStaticTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (18 ms)
[----------] 1 test from olLaunchKernelLocalMemStaticTest (18 ms total)

[----------] 1 test from olLaunchKernelGlobalCtorTest
[ RUN      ] olLaunchKernelGlobalCtorTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8bc15f8 (0x55f7f8bccb70))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8be0210, .ProgDataSize = 6224, .Program = 0x55f7f8bc1608 (0x55f7f8b98c70))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8b98c70, .Name = 0x55f7c3baf132 (global_ctor), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8bc1638 (0x55f7f8c53ee0))-> OL_SUCCESS
---> olMemAlloc(.Device = 0x55f7f8b74650, .Type = OL_ALLOC_TYPE_MANAGED, .Size = 256, .AllocationOut = 0x7ffc22734838 (0x204c00000))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8bccb70, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c53ee0, .ArgumentsData = 0x7ffc22734830, .ArgumentsSize = 8, .LaunchSizeArgs = 0x55f7f8bc1610 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8bccb70)-> OL_SUCCESS
---> olMemFree(.Address = 0x204c00000)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8b98c70)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8bccb70)-> OL_SUCCESS
[       OK ] olLaunchKernelGlobalCtorTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (37 ms)
[----------] 1 test from olLaunchKernelGlobalCtorTest (37 ms total)

[----------] 1 test from olLaunchKernelGlobalDtorTest
[ RUN      ] olLaunchKernelGlobalDtorTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8bf6ff8 (0x55f7f8bb0790))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734750 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc227346c4 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8c4ec70, .ProgDataSize = 2912, .Program = 0x55f7f8bf7008 (0x55f7f8bf7050))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8bf7050, .Name = 0x55f7c3baf13e (global_dtor), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8bf7038 (0x55f7f8bdb9d0))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8bb0790, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8bdb9d0, .ArgumentsData = nullptr, .ArgumentsSize = 0, .LaunchSizeArgs = 0x55f7f8bf7010 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8bb0790)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8bf7050)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8bb0790)-> OL_SUCCESS
[       OK ] olLaunchKernelGlobalDtorTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (19 ms)
[----------] 1 test from olLaunchKernelGlobalDtorTest (19 ms total)

[----------] 2 tests from olLaunchKernelGlobalTest
[ RUN      ] olLaunchKernelGlobalTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8bf4dc8 (0x55f7f8c5c340))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734710 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc22734684 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8c59e50, .ProgDataSize = 5492, .Program = 0x55f7f8bf4dd8 (0x55f7f8bdea90))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8bdea90, .Name = 0x55f7c3baf0c5 (write), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8bea740 (0x55f7f8c35cd0))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8bdea90, .Name = 0x55f7c3baf0cb (read), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8bea748 (0x55f7f8be4ba0))-> OL_SUCCESS
---> olMemAlloc(.Device = 0x55f7f8b74650, .Type = OL_ALLOC_TYPE_MANAGED, .Size = 256, .AllocationOut = 0x7ffc22734838 (0x204c00000))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8c5c340, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c35cd0, .ArgumentsData = nullptr, .ArgumentsSize = 0, .LaunchSizeArgs = 0x55f7f8bf4de0 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8c5c340)-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8c5c340, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8be4ba0, .ArgumentsData = 0x7ffc22734830, .ArgumentsSize = 8, .LaunchSizeArgs = 0x55f7f8bf4de0 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_SUCCESS
---> olWaitQueue(.Queue = 0x55f7f8c5c340)-> OL_SUCCESS
---> olMemFree(.Address = 0x204c00000)-> OL_SUCCESS
---> olDestroyProgram(.Program = 0x55f7f8bdea90)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8c5c340)-> OL_SUCCESS
[       OK ] olLaunchKernelGlobalTest.Success/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (18 ms)
[ RUN      ] olLaunchKernelGlobalTest.InvalidNotAKernel/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU
---> olCreateQueue(.Device = 0x55f7f8b74650, .Queue = 0x55f7f8bc9e68 (0x55f7f8c26490))-> OL_SUCCESS
---> olGetDeviceInfo(.Device = 0x55f7f8b74650, .PropName = OL_DEVICE_INFO_PLATFORM, .PropSize = 8, .PropValue = 0x7ffc22734710 (0x55f7f8331dc0))-> OL_SUCCESS
---> olGetPlatformInfo(.Platform = 0x55f7f8331dc0, .PropName = OL_PLATFORM_INFO_BACKEND, .PropSize = 4, .PropValue = 0x7ffc22734684 (OL_PLATFORM_BACKEND_CUDA))-> OL_SUCCESS
---> olCreateProgram(.Device = 0x55f7f8b74650, .ProgData = 0x55f7f8bc1390, .ProgDataSize = 5492, .Program = 0x55f7f8bc9e78 (0x55f7f8c35cd0))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8c35cd0, .Name = 0x55f7c3baf0c5 (write), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8bea740 (0x55f7f8c36860))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8c35cd0, .Name = 0x55f7c3baf0cb (read), .Kind = OL_SYMBOL_KIND_KERNEL, .Symbol = 0x55f7f8bea748 (0x55f7f8c366e0))-> OL_SUCCESS
---> olGetSymbol(.Program = 0x55f7f8c35cd0, .Name = 0x55f7c3badfa2 (global), .Kind = OL_SYMBOL_KIND_GLOBAL_VARIABLE, .Symbol = 0x7ffc22734848 (0x55f7f8c06750))-> OL_SUCCESS
---> olLaunchKernel(.Queue = 0x55f7f8c26490, .Device = 0x55f7f8b74650, .Kernel = 0x55f7f8c06750, .ArgumentsData = nullptr, .ArgumentsSize = 0, .LaunchSizeArgs = 0x55f7f8bc9e80 ((struct ol_kernel_launch_size_args_t){.Dimensions = 1, .NumGroups = (struct ol_dimensions_t){.x = 1, .y = 1, .z = 1}, .GroupSize = (struct ol_dimensions_t){.x = 64, .y = 1, .z = 1}, .DynSharedMemory = 0}), .EventOut = nullptr)-> OL_ERRC_SYMBOL_KIND
     *Error Details* provided symbol is not a kernel
---> olDestroyProgram(.Program = 0x55f7f8c35cd0)-> OL_SUCCESS
---> olDestroyQueue(.Queue = 0x55f7f8c26490)-> OL_SUCCESS
[       OK ] olLaunchKernelGlobalTest.InvalidNotAKernel/CUDA_NVIDIA_GeForce_RTX_4070_Laptop_GPU (15 ms)
[----------] 2 tests from olLaunchKernelGlobalTest (34 ms total)

[----------] Global test environment tear-down
[==========] 10 tests from 8 test suites ran. (195 ms total)
[  PASSED  ] 10 tests.
---> olShutDownkernel.unittests: /home/leandro/llvm-project/offload/plugins-nextgen/common/src/PluginInterface.cpp:1658: Error llvm::omp::target::plugin::GenericPluginTy::deinit(): Assertion `Initialized && "Plugin was not initialized!"' failed.
pure virtual method called
terminate called without an active exception
Aborted (core dumped)

[leandrolcampos]

@callumfare, I also enabled assertions when I built the Offload:

cd llvm-project
cmake -S llvm -B build -G Ninja \
  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld" \
  -DLLVM_ENABLE_RUNTIMES="openmp;offload;libc" \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_PARALLEL_LINK_JOBS=1 \
  -DCMAKE_INSTALL_PREFIX="$LLVM_HOME" \
  -DRUNTIMES_nvptx64-nvidia-cuda_LLVM_ENABLE_RUNTIMES=libc \
  -DLLVM_RUNTIME_TARGETS="default;nvptx64-nvidia-cuda"

[callumfare]

@leandrolcampos Thanks, didn't realise I had assertions disabled, I can reproduce this locally now. I'll investigate the issue.

[callumfare]

I've put up a quick fix for the issue in #148642, although I think a follow up is needed to generally prevent problems tearing down.

I've also spotted another issue where some kernel tests are sporadically failing when running kernel.unittests directly, which I don't see with the check-offload-unit target. I'll investigate those failures.