'onnxruntime_providers_cuda.dll' crashes with access violation during TLS callback initialization (v1.22.1, CUDA 12.9, Windows)

[xgirones]

Describe the issue

Summary

The CUDA execution provider DLL crashes with an access violation (0xC0000005) when attempting to load it on Windows, either via LoadLibrary() or through the official ONNX Runtime API. The crash occurs during Thread Local Storage (TLS) callback initialization, specifically when the second TLS callback attempts to read from a null pointer (0x0000000000000000).

Environment

Field	Value
ONNX Runtime Version	1.22.1 (built from source)
Operating System	Windows 10.0.19045 Build 19045
CUDA Version	12.9
cuDNN Version	9.11.0.98 for CUDA 12
Visual Studio Version	Visual Studio 2022 v17.14.6 (June 2025)
CMake Version	3.30.0
Build Configuration	Release

Steps to Reproduce

1. Build ONNX Runtime from source using the following command:

build.bat --cmake_path "C:\Program Files\CMake\bin\cmake.exe" ^
    --config Release ^
    --use_cuda ^
    --cuda_version 12.9 ^
    --build_shared_lib ^
    --parallel ^
    --compile_no_warning_as_error ^
    --cudnn_home D:\ThirdPartyLib\cudann\cudnn-windows-x86_64-9.11.0.98_cuda12 ^
    --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9" ^
    --skip_tests ^
    --cmake_generator "Visual Studio 17 2022"

2. Copy the generated DLLs to the application directory:

   • onnxruntime.dll
   • onnxruntime_providers_shared.dll
   • onnxruntime_providers_cuda.dll

3. Create a minimal test application:

#include <windows.h>
#include <iostream>

int main() {
    HMODULE hModule = LoadLibrary(_T("onnxruntime_providers_cuda.dll"));
    if (hModule == NULL) {
        DWORD error = GetLastError();
        std::cerr << "LoadLibrary failed with error: 0x" << std::hex << error << std::endl;
        WCHAR errorMsg[256];
        FormatMessageW(FORMAT_MESSAGE_FROM_SYSTEM, NULL, error, 0, errorMsg, 256, NULL);
        std::wcout << L"Error message: " << errorMsg << std::endl;
    } else {
        std::cout << "DLL loaded successfully" << std::endl;
        FreeLibrary(hModule);
    }
    return 0;
}

4. Run the application - it crashes immediately during DLL initialization

Expected Behavior

The DLL should load successfully without crashing, allowing the CUDA execution provider to be used for inference acceleration.

Actual Behavior

The application crashes with an unhandled exception during the DLL's TLS callback initialization:

Exception thrown at 0x00007FF982E66931 (onnxruntime_providers_cuda.dll): 0xC0000005: 
Access violation reading location 0x0000000000000000.

Error Output / Stack Trace

Debugger Output

Exception thrown at 0x00007FF982E66931 (onnxruntime_providers_cuda.dll) in workbench_gpu.exe: 
0xC0000005: Access violation reading location 0x0000000000000000.

Unhandled exception at 0x00007FF982E66931 (onnxruntime_providers_cuda.dll) in workbench_gpu.exe: 
0xC0000005: Access violation reading location 0x0000000000000000.

Loader Trace (via gflags.exe)

LdrpInitializeNode - INFO: Calling init routine 00007FF982F7ED1C for DLL "onnxruntime_providers_cuda.dll"
LdrpCallTlsInitializers - INFO: Calling TLS callback 00007FF982F7DECC for DLL "onnxruntime_providers_cuda.dll" at 00007FF9825E0000
LdrpCallTlsInitializers - INFO: Calling TLS callback 00007FF982F7E600 for DLL "onnxruntime_providers_cuda.dll" at 00007FF9825E0000
Exception thrown at 0x00007FF982E66931 (onnxruntime_providers_cuda.dll): 0xC0000005: Access violation reading location 0x0000000000000000.

Additional Information

What I've Tried

• ✅ Verified all CUDA and cuDNN dependencies are present and accessible via PATH
• ✅ Tested with both Debug and Release builds - same crash
• ✅ Pre-initialized CUDA context before loading (cudaSetDevice(0); cudaFree(0);) - didn't help
• ✅ Loaded DLLs in dependency order (onnxruntime.dll → onnxruntime_providers_shared.dll → onnxruntime_providers_cuda.dll)
• ✅ Used official ONNX Runtime C API for initialization - crashes the same way
• ✅ Verified there is only a single CUDA installation on system
• ✅ Confirmed NVIDIA driver supports CUDA 12.9 via nvidia-smi

Analysis

The crash occurs specifically in the second TLS callback.

Impact

This issue completely prevents the use of CUDA 12.9 acceleration with ONNX Runtime on Windows when building from source.

Dependencies

snap_trace_tail.txt

Urgency

This issue prevents the use of CUDA 12.9 with ONNX Runtime on Windows when building from source.

Target platform

Windows

Build script

build.bat --cmake_path "C:\Program Files\CMake\bin\cmake.exe" ^
    --config Release ^
    --use_cuda ^
    --cuda_version 12.9 ^
    --build_shared_lib ^
    --parallel ^
    --compile_no_warning_as_error ^
    --cudnn_home D:\ThirdPartyLib\cudann\cudnn-windows-x86_64-9.11.0.98_cuda12 ^
    --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9" ^
    --skip_tests ^
    --cmake_generator "Visual Studio 17 2022"

Error / output

Debugger Output

Exception thrown at 0x00007FF982E66931 (onnxruntime_providers_cuda.dll) in workbench_gpu.exe: 
0xC0000005: Access violation reading location 0x0000000000000000.

Unhandled exception at 0x00007FF982E66931 (onnxruntime_providers_cuda.dll) in workbench_gpu.exe: 
0xC0000005: Access violation reading location 0x0000000000000000.

Loader Trace (via gflags.exe)

LdrpInitializeNode - INFO: Calling init routine 00007FF982F7ED1C for DLL "onnxruntime_providers_cuda.dll"
LdrpCallTlsInitializers - INFO: Calling TLS callback 00007FF982F7DECC for DLL "onnxruntime_providers_cuda.dll" at 00007FF9825E0000
LdrpCallTlsInitializers - INFO: Calling TLS callback 00007FF982F7E600 for DLL "onnxruntime_providers_cuda.dll" at 00007FF9825E0000
Exception thrown at 0x00007FF982E66931 (onnxruntime_providers_cuda.dll): 0xC0000005: Access violation reading location 0x0000000000000000.

Visual Studio Version

Visual Studio 2022 v17.14.6 (June 2025)

GCC / Compiler Version

No response

[yuslepukhin]

You can not free load the cuda EP library. It must be loaded from onnxruntime and one more library is loaded with it that sets necessary interface pointers.

[xgirones]

Hi Dmitri,

Thank you for the clarification about not being able to directly load the CUDA EP library. I've now switched to using the ONNX Runtime API for initialization, but I'm still encountering a failure when trying to initialize the CUDA provider.

Current Approach (Following Your Guidance)

I'm now using the official ONNX Runtime C API to initialize the CUDA provider:

OrtEnv* env;
OrtCreateEnv(ORT_LOGGING_LEVEL_VERBOSE, "test", &env);

OrtSessionOptions* session_options;
OrtCreateSessionOptions(&session_options);

...

// This call triggers the error
OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, &cuda_options);

New Error

When OrtSessionOptionsAppendExecutionProvider_CUDA is called, Windows displays a "Bad Image" error dialog:

workbench.exe - Bad Image
d:\projects\simplelpr_port.git\code\output\visual_studio\2022\bin\x64\Release\onnxruntime_providers_cuda.dll 
is either not designed to run on Windows or it contains an error. 
Error status 0xc000007b.

The ONNX Runtime logs show:

[E:onnxruntime:test, provider_bridge_ort.cc:1938] onnxruntime::ProviderLibrary::Get 
[ONNXRuntimeError] : 1 : FAIL : Error loading "...\onnxruntime_providers_cuda.dll" which is missing. 
(Error 193: "%1 is not a valid Win32 application.")

Diagnostic Steps Completed

I've verified that:

• ✅ All binaries (exe, onnxruntime.dll, onnxruntime_providers_cuda.dll) are x64
• ✅ Both ONNX Runtime and my application are built in Release mode (also tried Debug)
• ✅ All CUDA/cuDNN dependencies are present and accessible via PATH
• ✅ onnxruntime_providers_shared.dll is in the same directory
• ✅ Dependencies.exe shows no missing dependencies
• ✅ dumpbin /headers shows valid PE headers for all DLLs
• ✅ CUDA runtime DLLs are found (where cudart64_12.dll succeeds)

Environment

• Built from source: v1.22.1
• Build command as shown in original issue
• Windows 10 x64, VS2022, CUDA 12.9, cuDNN 9.11.0.98

Question

The error suggests the CUDA provider DLL has an invalid format, but all diagnostics show it should be valid. Is there a specific build configuration or additional dependency required for the CUDA provider to load properly through the ONNX Runtime API?

Any guidance on what might cause this "Bad Image" error when loading through the proper API would be greatly appreciated.

Thank you!