CUDA driver version is insufficient for CUDA runtime version

[mausch]

I was going to submit an issue for this but it's likely not really an issue with ONNX but rather an issue with the unmitigated disaster that is still trying to work with CUDA.
Any ideas?

Describe the bug

Attempting to optimise a model with ORTOptimizer throws this exception:

Traceback (most recent call last):
...
  File "/home/ubuntu/elevate.jobtitles/.mamba/envs/jobtitles/lib/python3.7/site-packages/optimum/onnxruntime/optimization.py", line 142, in export
    only_onnxruntime=optimization_config.optimize_with_onnxruntime_only,
  File "/home/ubuntu/elevate.jobtitles/.mamba/envs/jobtitles/lib/python3.7/site-packages/onnxruntime/transformers/optimizer.py", line 238, in optimize_model
    disabled_optimizers=disabled_optimizers,
  File "/home/ubuntu/elevate.jobtitles/.mamba/envs/jobtitles/lib/python3.7/site-packages/onnxruntime/transformers/optimizer.py", line 103, in optimize_by_onnxruntime
    onnx_model_path, sess_options, providers=["CUDAExecutionProvider"], **kwargs
  File "/home/ubuntu/elevate.jobtitles/.mamba/envs/jobtitles/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 347, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/ubuntu/elevate.jobtitles/.mamba/envs/jobtitles/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 395, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
RuntimeError: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:122 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] CUDA failure 35: CUDA driver version is insufficient for CUDA runtime version ; GPU=0 ; hostname=ip-172-31-33-171 ; expr=cudaSetDevice(info_.device_id);

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 22.04
• ONNX Runtime installed from (source or binary): pip
• ONNX Runtime version:

$ pip list | grep onnx
onnx                          1.12.0
onnxruntime                   1.12.1
onnxruntime-gpu               1.12.1

• Python version: 3.7.12
• CUDA/cuDNN version:

$ micromamba list | grep cud
  cudatoolkit                    11.7.0     hd8887f6_10           conda-forge/linux-64
  cudnn                          8.4.1.50   hed8a83a_0            conda-forge/linux-64

• GPU model and memory:

n$ nvidia-smi
Thu Sep  1 15:59:21 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A10G         On   | 00000000:00:1E.0 Off |                    0 |
|  0%   32C    P8    23W / 300W |      0MiB / 23028MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

I'm sure some version of something somewhere is not quite what something else was expecting.
It's just ridiculous that in 2022 working with CUDA is still such a mess.
According to https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements "Onnx Runtime built with CUDA 11.4 should be compatible with any CUDA 11.x version" so this should work? CUDA 11.4 is no longer even available from nvidia's ubuntu repo.

Many thanks.