Failed to load library libonnxruntime_providers_cuda.so

[InsaneGe]

Firstly, I follow this link install gpu-version sherpa-onnx, "ImportError: /lib64/libm.so.6: version GLIBC_2.27 not found" happens, then I follow this link to solve it and pass the test python3 -c "import sherpa_onnx; print(sherpa_onnx.__file__)"
but when I run python python-api-examples/offline-fire-red-asr-decode-files.py, the following error happens, I have no idea how to solve it, look forward to your help🙏!

Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/sherpa-onnx/python-api-examples/offline-fire-red-asr-decode-files.py", line 97, in <module>
    main()
  File "/home/ec2-user/SageMaker/sherpa-onnx/python-api-examples/offline-fire-red-asr-decode-files.py", line 69, in main
    recognizer, wave_filename = create_recognizer()
                                ^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/SageMaker/sherpa-onnx/python-api-examples/offline-fire-red-asr-decode-files.py", line 57, in create_recognizer
    sherpa_onnx.OfflineRecognizer.from_fire_red_asr(
  File "/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/offline_recognizer.py", line 643, in from_fire_red_asr
    self.recognizer = _Recognizer(recognizer_config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1193 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

my environment is as follows:

(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ which nvcc
/usr/local/cuda-12.1/bin/nvcc
(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ find /usr/local/cuda/ -name libcudnn.so*
/usr/local/cuda/lib/libcudnn.so
/usr/local/cuda/lib/libcudnn.so.8
/usr/local/cuda/lib/libcudnn.so.8.9.3
(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ nvidia-smi
Mon Jul  7 06:35:04 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
|  0%   33C    P0             69W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ python3 -c "import sherpa_onnx; print(sherpa_onnx.__file__)"
/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/__init__.py

[InsaneGe]

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16.tar.bz2
by the way, Is there any documentation explaining how the sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16 model in the link was created?

[csukuangfj]

then I follow #1586 to solve it and pass the test python3 -c "import sherpa_onnx; print(sherpa_onnx.file)"

please post its output

[csukuangfj]

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16.tar.bz2 by the way, Is there any documentation explaining how the sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16 model in the link was created?

please see FireRedTeam/FireRedASR#23 (comment)

[InsaneGe]

(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ python3 -c "import sherpa_onnx; print(sherpa_onnx.__file__)"
/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/__init__.py

BTW, we only encounter this problem when using AWS platform, we can run this following python script successfully in Ubuntu22.04.1 platform. we want to know how to run in AWS platform.

import time
from pathlib import Path
import pynvml

import sherpa_onnx
import soundfile as sf
from contextlib import contextmanager
import torch
@contextmanager
def measure_gpu_memory_torch():
    '''
    当使用 provider='cuda' 时，ONNX Runtime直接使用CUDA API分配显存，
    PyTorch的 torch.cuda.max_memory_allocated() 只能统计通过PyTorch分配的显存
    '''
    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats()
    yield
    peak_memory = torch.cuda.max_memory_allocated()
    print(f"the max memory usage in onnx reasoning process: {peak_memory / (1024 ** 2):.2f} MB")
@contextmanager
def measure_gpu_memory():
    """Context manager to measure peak GPU memory usage using nvidia-ml-py."""
    pynvml.nvmlInit()
    handle = pynvml.nvmlDeviceGetHandleByIndex(0)

    initial_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
    initial_used = float(initial_info.used)
    
    yield
    
    peak_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
    peak_used = float(peak_info.used)
    
    memory_increase = peak_used - initial_used
    print(f"increase memory: {memory_increase / (1024 ** 3):.3f} GB")
    print(f"max inference memory: {peak_used / (1024 ** 3):.3f} GB")


def create_recognizer():
    encoder = "./sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16/encoder.int8.onnx"
    decoder = "./sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16/decoder.int8.onnx"
    tokens = "./sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16/tokens.txt"
    test_wav = "./sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16/test_wavs/0.wav"

    if (
        not Path(encoder).is_file()
        or not Path(decoder).is_file()
        or not Path(test_wav).is_file()
    ):
        raise ValueError(
            """Please download model files from
            https://github.com/k2-fsa/sherpa-onnx/releases/tag/asr-models
            """
        )
    return (
        sherpa_onnx.OfflineRecognizer.from_fire_red_asr(
            encoder=encoder,
            decoder=decoder,
            tokens=tokens,
            debug=True,
            provider='cuda'
        ),
        test_wav,
    )


def main():
    with measure_gpu_memory():
        recognizer, wave_filename = create_recognizer()
        audio, sample_rate = sf.read(wave_filename, dtype="float32", always_2d=True)
        audio = audio[:, 0]  # only use the first channel

        # audio is a 1-D float32 numpy array normalized to the range [-1, 1]
        # sample_rate does not need to be 16000 Hz
        stream = recognizer.create_stream()
        stream.accept_waveform(sample_rate, audio)
        start_time = time.time()
        recognizer.decode_stream(stream)
        end_time = time.time()
    inference_time_ms = (end_time - start_time) * 1000
    print(f"time of onnx reasoning: {inference_time_ms:.2f} ms")
    
    print(wave_filename)
    print(stream.result)

if __name__ == "__main__":
    main()

what is more, the time consumed isn't ideal when we run successfully in Ubuntu22.04.01.
reason the same audio file in the same GPU:
when setting provider = "cuda", time of onnx reasoning: 6279.24 ms
when setting provider = "cpu", time of onnx reasoning: 20405.59 ms
but we directly run the official uncompressed model, it only takes 2883.39 ms,(the script as follows), I really want to know why it become slower🙏

from fireredasr.models.fireredasr import FireRedAsr
import time
import torch
from contextlib import contextmanager
@contextmanager
def measure_gpu_memory():
    """Context manager to measure peak GPU memory usage."""
    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats()
    yield
    peak_memory = torch.cuda.max_memory_allocated()
    print(f"max memory usage: {peak_memory / (1024 ** 3):.2f} GB")

batch_uttid = ["BAC009S0764W0121"]
batch_wav_path = ["/home/ec2-user/SageMaker/FireRedASR/examples/wav/BAC009S0764W0121.wav"]
batch_wav_path = ["/root/sherpa-onnx/sherpa-onnx-fire-red-asr-large-zh_en-2025-02-16/test_wavs/0.wav"]

# FireRedASR-AED
model = FireRedAsr.from_pretrained("aed", "/root/FireRedTeam/FireRedASR-AED-L")
start_time = time.time()
with measure_gpu_memory():
    results = model.transcribe(
        batch_uttid,
        batch_wav_path,
        {
            "use_gpu": 1,
            "beam_size": 3,
            "nbest": 1,
            "decode_max_len": 0,
            "softmax_smoothing": 1.25,
            "aed_length_penalty": 0.6,
            "eos_penalty": 1.0
        }
    )
    end_time = time.time()
print(results)
inference_time_ms = (end_time - start_time) * 1000
print(f"asr inference time: {inference_time_ms:.2f} ms")

[csukuangfj]

find /home/ec2-user/anaconda3/envs/sherpa/ -name libonnxruntime_providers_cuda.so

what is the output of the above command?

[InsaneGe]

find /home/ec2-user/anaconda3/envs/sherpa/ -name libonnxruntime_providers_cuda.so

what is the output of the above command?

(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ find /home/ec2-user/anaconda3/envs/sherpa/ -name libonnxruntime_providers_cuda.so
/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/lib/libonnxruntime_providers_cuda.so

[csukuangfj]

export LD_LIBRARY_PATH=/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/lib:$LD_LIBRARY_PATH

After adding it, can you re-try?

If you still get the same error, please post

echo $LD_LIBRARY_PATH

[InsaneGe]

export LD_LIBRARY_PATH=/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/lib:$LD_LIBRARY_PATH
I've try this before, it can't work

(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ export LD_LIBRARY_PATH=/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/lib:$LD_LIBRARY_PATH
(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ python python-api-examples/offline-fire-red-asr-decode-files.py 
Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/sherpa-onnx/python-api-examples/offline-fire-red-asr-decode-files.py", line 94, in <module>
    main()
  File "/home/ec2-user/SageMaker/sherpa-onnx/python-api-examples/offline-fire-red-asr-decode-files.py", line 69, in main
    recognizer, wave_filename = create_recognizer()
                                ^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/SageMaker/sherpa-onnx/python-api-examples/offline-fire-red-asr-decode-files.py", line 57, in create_recognizer
    sherpa_onnx.OfflineRecognizer.from_fire_red_asr(
  File "/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/offline_recognizer.py", line 643, in from_fire_red_asr
    self.recognizer = _Recognizer(recognizer_config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1193 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ echo $LD_LIBRARY_PATH
/home/ec2-user/anaconda3/envs/sherpa/lib/python3.12/site-packages/sherpa_onnx/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/lib:/usr/lib:/lib
(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ 

[csukuangfj]

libcublasLt.so.11:

You need to install cudalltoolkit with cudnn

Please refer to our installation doc. It has a link to k2's doc.

[csukuangfj]

Make sure you have installed the correct version of cudatoolkit as described in the doc.

[csukuangfj]

You need to pay extra effort to use GPU.

[InsaneGe]

libcublasLt.so.11:

You need to install cudalltoolkit with cudnn

Please refer to our installation doc. It has a link to k2's doc.

but I've installed cudnn already by this link, as u can see below

(sherpa) ec2-user@ip-172-16-75-115:~/SageMaker/sherpa-onnx$ find /usr/local/cuda/ -name libcudnn.so*
/usr/local/cuda/lib/libcudnn.so
/usr/local/cuda/lib/libcudnn.so.8
/usr/local/cuda/lib/libcudnn.so.8.9.3

And can u reply to me do u know why it become slower by using the compressed onnx model?

when setting provider = "cuda", time of onnx reasoning: 6279.24 ms
when setting provider = "cpu", time of onnx reasoning: 20405.59 ms
but we directly run the official uncompressed model, it only takes 2883.39 ms,(the script as follows)

[csukuangfj]

RuntimeError: /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1193 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

If you have installed cudnn correctly, please tell us where is the file libcublasLt.so.11 on your system.

If you cannot find it, please follow strictly our doc.

[csukuangfj]

but I've installed cudnn already by this link,

Can you tell us why you select cuda 12.1?

Can you follow the version described in the sherpa-onnx doc?

[InsaneGe]

but I've installed cudnn already by this link,

Can you tell us why you select cuda 12.1?

Can you follow the version described in the sherpa-onnx doc?

ok, I've fixed this bug by following link thanks a lot!
but I really want to ask again do u have some ideas about why it becomes slower by using onnx-fireredasr model than original model.