[Bug]: Impossible to install with cuda 13 on linux

[skier233]

System Info

# Install torch/torchvision
pip install torch==2.9.0 torchvision --index-url https://download.pytorch.org/whl/cu130

python - <<'PY'
import torch, os
print("torch", torch.__version__, "built with CUDA", torch.version.cuda)
print("LD_LIBRARY_PATH =", os.environ.get("LD_LIBRARY_PATH"))
PY

-->

torch 2.9.0+cu130 built with CUDA 13.0
LD_LIBRARY_PATH = None

pip install --extra-index-url https://pypi.nvidia.com tensorrt-llm==1.2.0rc2

This installs CUDA 13.0. But it downgrades pytorch to cuda 12.8:

python - <<'PY'
import torch, os
print("torch", torch.__version__, "built with CUDA", torch.version.cuda)
print("LD_LIBRARY_PATH =", os.environ.get("LD_LIBRARY_PATH"))
PY

-->

torch 2.9.0+cu128 built with CUDA 12.8
LD_LIBRARY_PATH = None

This then makes it impossible to use because torch is now expecting cuda 12.8 but tensorrt-llm has overwritten the underlying cuda to be 13.0

I see this manifest in this error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/mnt/c/Coding/Testing/PyTorch/MultiClassImageClassification/src/modern_imclas/export_tensorrt.py", line 307, in <module>
    main()
  File "/mnt/c/Coding/Testing/PyTorch/MultiClassImageClassification/src/modern_imclas/export_tensorrt.py", line 294, in main
    compile_with_torch_trt(
  File "/mnt/c/Coding/Testing/PyTorch/MultiClassImageClassification/src/modern_imclas/export_tensorrt.py", line 200, in compile_with_torch_trt
    import torch_tensorrt  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/tyler/miniconda3/envs/new_training_env/lib/python3.12/site-packages/torch_tensorrt/__init__.py", line 79, in <module>
    _register_with_torch()
  File "/home/tyler/miniconda3/envs/new_training_env/lib/python3.12/site-packages/torch_tensorrt/__init__.py", line 66, in _register_with_torch
    torch.ops.load_library(linked_file_full_path)
  File "/home/tyler/miniconda3/envs/new_training_env/lib/python3.12/site-packages/torch/_ops.py", line 1392, in load_library
    ctypes.CDLL(path)
  File "/home/tyler/miniconda3/envs/new_training_env/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libcudart.so.13: cannot open shared object file: No such file or directory

I have not been able to find any version combination of cuda+pytorch+tensorrt_llm that will work with pytorch, torchvision, torch-tensorrt, cuda, tensorrt_llm.

Who can help?

@kaiyux

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

# Install torch/torchvision
pip install torch==2.9.0 torchvision --index-url https://download.pytorch.org/whl/cu130

python - <<'PY'
import torch, os
print("torch", torch.__version__, "built with CUDA", torch.version.cuda)
print("LD_LIBRARY_PATH =", os.environ.get("LD_LIBRARY_PATH"))
PY

-->

torch 2.9.0+cu130 built with CUDA 13.0
LD_LIBRARY_PATH = None

pip install --extra-index-url https://pypi.nvidia.com tensorrt-llm==1.2.0rc2

This installs CUDA 13.0. But it downgrades pytorch to cuda 12.8:

python - <<'PY'
import torch, os
print("torch", torch.__version__, "built with CUDA", torch.version.cuda)
print("LD_LIBRARY_PATH =", os.environ.get("LD_LIBRARY_PATH"))
PY

-->

torch 2.9.0+cu128 built with CUDA 12.8
LD_LIBRARY_PATH = None

This then makes it impossible to use because torch is now expecting cuda 12.8 but tensorrt-llm has overwritten the underlying cuda to be 13.0

Expected behavior

the cuda version should align

actual behavior

cuda version and pytorch expected cuda version are misaligned thus making the release unusable

additional notes

n/a

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[pcastonguay]

@chzblych can you help? Thanks.

[yanzhibo-bytedance]

have you solved this problem?

[AaronCaoZJ]

same problem, have you solved it？

[skier233]

It doesn’t seem possible to solve until someone from Nvidia gets a chance to fix this. @chzblych can you take a look?

[thad0ctor]

I think I got around it locally by updating requirements.txt as follows.

Remove:

torch>=2.9.0a0,<=2.9.0
torchvision

Replace with:

# Use PyTorch CUDA 13 wheels from PyTorch index (PEP 508 direct URL references)
# This ensures pip installs the CUDA 13 version instead of defaulting to CUDA 12.x from pypi.org
# See https://github.com/NVIDIA/TensorRT-LLM/issues/9318 for context on this workaround

# x86_64 Python 3.10
torch @ https://download.pytorch.org/whl/cu130/torch-2.9.0+cu130-cp310-cp310-manylinux_2_28_x86_64.whl ; python_version >= "3.10" and python_version < "3.11" and platform_machine == "x86_64" and platform_system == "Linux"
torchvision @ https://download.pytorch.org/whl/cu130/torchvision-0.24.0+cu130-cp310-cp310-manylinux_2_28_x86_64.whl ; python_version >= "3.10" and python_version < "3.11" and platform_machine == "x86_64" and platform_system == "Linux"

# x86_64 Python 3.12
torch @ https://download.pytorch.org/whl/cu130/torch-2.9.0+cu130-cp312-cp312-manylinux_2_28_x86_64.whl ; python_version >= "3.12" and python_version < "3.13" and platform_machine == "x86_64" and platform_system == "Linux"
torchvision @ https://download.pytorch.org/whl/cu130/torchvision-0.24.0+cu130-cp312-cp312-manylinux_2_28_x86_64.whl ; python_version >= "3.12" and python_version < "3.13" and platform_machine == "x86_64" and platform_system == "Linux"

# aarch64 Python 3.10
torch @ https://download.pytorch.org/whl/cu130/torch-2.9.0+cu130-cp310-cp310-manylinux_2_28_aarch64.whl ; python_version >= "3.10" and python_version < "3.11" and platform_machine == "aarch64" and platform_system == "Linux"
torchvision ; python_version >= "3.10" and python_version < "3.11" and platform_machine == "aarch64" and platform_system == "Linux"

# aarch64 Python 3.12
torch @ https://download.pytorch.org/whl/cu130/torch-2.9.0+cu130-cp312-cp312-manylinux_2_28_aarch64.whl ; python_version >= "3.12" and python_version < "3.13" and platform_machine == "aarch64" and platform_system == "Linux"
torchvision ; python_version >= "3.12" and python_version < "3.13" and platform_machine == "aarch64" and platform_system == "Linux"

Verification:

echo "=== Verification ==="

python3 -c "
import torch, tensorrt as trt, tensorrt_llm, transformers, safetensors, modelopt
print(f'PyTorch {torch.__version__} + CUDA {torch.version.cuda}')
print(f'TensorRT {trt.__version__}')
print(f'TensorRT-LLM {tensorrt_llm.__version__}')
print(f'transformers {transformers.__version__}')
print(f'safetensors {safetensors.__version__}')
print('nvidia-modelopt OK')
"

Result:
PyTorch 2.9.0+cu130 + CUDA 13.0
TensorRT 10.14.1.48
TensorRT-LLM 1.2.0rc6
transformers 4.57.1
safetensors 0.7.0
nvidia-modelopt OK

I am able to convert models and interface with a TensorRT engine using this fix. Performance is mixed and may depend on engine configuration.

[thad0ctor]

Bump, any traction on this? I outlined a potential solution above. Thank you.

@chzblych @kaiyux @pcastonguay

[ZhanruiSunCh]

Prevent pip from replacing existing PyTorch installation

On certain systems, particularly Ubuntu 22.04, users installing TensorRT LLM would find that their existing, CUDA 13.0 compatible PyTorch installation (e.g., torch==2.9.0+cu130) was being uninstalled by pip. It was then replaced by a CUDA 12.8 version (torch==2.9.0), causing the TensorRT LLM installation to be unusable and leading to runtime errors.

The solution is to create a pip constraints file, locking torch to the currently installed version. Here is an example of how this can be done manually:

CURRENT_TORCH_VERSION=$(python3 -c "import torch; print(torch.__version__)")
echo "torch==$CURRENT_TORCH_VERSION" > /tmp/torch-constraint.txt
pip3 install --upgrade pip setuptools && pip3 install tensorrt_llm -c /tmp/torch-constraint.txt

https://nvidia.github.io/TensorRT-LLM/latest/installation/linux.html