TensorRT hangs

[Notbici]

System Info

I run this:
trtllm-serve /tensorstuff/TensorRT-Model-Optimizer/examples/llm_ptq/saved_models_qwen3-32b_nvfp4_hf/ --host --port 8000 --backend pytorch

It outputs this:

--backend pytorch
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
2025-06-28 17:32:17,461 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.20.0rc3
[06/28/2025-17:32:17] [TRT-LLM] [I] Compute capability: (12, 0)
[06/28/2025-17:32:17] [TRT-LLM] [I] SM count: 170
[06/28/2025-17:32:17] [TRT-LLM] [I] SM clock: 3105 MHz
[06/28/2025-17:32:17] [TRT-LLM] [I] int4 TFLOPS: 0
[06/28/2025-17:32:17] [TRT-LLM] [I] int8 TFLOPS: 0
[06/28/2025-17:32:17] [TRT-LLM] [I] fp8 TFLOPS: 0
[06/28/2025-17:32:17] [TRT-LLM] [I] float16 TFLOPS: 0
[06/28/2025-17:32:17] [TRT-LLM] [I] bfloat16 TFLOPS: 0
[06/28/2025-17:32:17] [TRT-LLM] [I] float32 TFLOPS: 0
[06/28/2025-17:32:17] [TRT-LLM] [I] Total Memory: 31 GiB
[06/28/2025-17:32:17] [TRT-LLM] [I] Memory clock: 14001 MHz
[06/28/2025-17:32:17] [TRT-LLM] [I] Memory bus width: 512
[06/28/2025-17:32:17] [TRT-LLM] [I] Memory bandwidth: 1792 GB/s
[06/28/2025-17:32:17] [TRT-LLM] [I] PCIe speed: 2500 Mbps
[06/28/2025-17:32:17] [TRT-LLM] [I] PCIe link width: 8
[06/28/2025-17:32:17] [TRT-LLM] [I] PCIe bandwidth: 2 GB/s
[06/28/2025-17:32:17] [TRT-LLM] [I] Set nccl_plugin to None.
[06/28/2025-17:32:17] [TRT-LLM] [I] Found /tensorstuff/TensorRT-Model-Optimizer/examples/llm_ptq/saved_models_qwen3-32b_nvfp4_hf/hf_quant_config.json, pre-quantized checkpoint is used.
[06/28/2025-17:32:17] [TRT-LLM] [I] Setting quant_algo=NVFP4 form HF quant config.
[06/28/2025-17:32:17] [TRT-LLM] [I] Setting kv_cache_quant_algo=FP8 form HF quant config.
[06/28/2025-17:32:17] [TRT-LLM] [I] Setting group_size=16 from HF quant config.
[06/28/2025-17:32:17] [TRT-LLM] [I] Setting exclude_modules=['lm_head'] from HF quant config.
[06/28/2025-17:32:18] [TRT-LLM] [I] PyTorchConfig(extra_resource_managers={}, use_cuda_graph=False, cuda_graph_batch_sizes=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 64, 128], cuda_graph_max_batch_size=128, cuda_graph_padding_enabled=False, disable_overlap_scheduler=False, moe_max_num_tokens=None, attn_backend='TRTLLM', moe_backend='CUTLASS', mixed_sampler=False, enable_trtllm_sampler=False, kv_cache_dtype='auto', use_kv_cache=True, enable_iter_perf_stats=False, enable_iter_req_stats=False, print_iter_log=False, torch_compile_enabled=False, torch_compile_fullgraph=True, torch_compile_inductor_enabled=False, torch_compile_piecewise_cuda_graph=False, torch_compile_enable_userbuffers=True, autotuner_enabled=True, enable_layerwise_nvtx_marker=False, load_format=<LoadFormat.AUTO: 0>)
rank 0 using MpiPoolSession to spawn MPI processes
[06/28/2025-17:32:18] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_queue
[06/28/2025-17:32:18] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_error_queue
[06/28/2025-17:32:18] [TRT-LLM] [I] Generating a new HMAC key for server proxy_result_queue
[06/28/2025-17:32:18] [TRT-LLM] [I] Generating a new HMAC key for server proxy_stats_queue
[06/28/2025-17:32:18] [TRT-LLM] [I] Generating a new HMAC key for server proxy_kv_cache_events_queue

Hanging at that bit.

I see 577MB of vram allocated to GPU 1 thats it, no further action for hours.

CPU shows 1 to 2 cores firing off every now and then. I tried a --verbose flag but I cannot get more information than this.

NVIDIA-SMI

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.169                Driver Version: 570.169        CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        On  |   00000000:21:00.0 Off |                  N/A |
|  0%   26C    P8             13W /  575W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5090        On  |   00000000:22:00.0 Off |                  N/A |
|  0%   27C    P8             12W /  575W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 5090        On  |   00000000:61:00.0 Off |                  N/A |
|  0%   26C    P8              3W /  575W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 5090        On  |   00000000:62:00.0 Off |                  N/A |
|  0%   27C    P8              4W /  575W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run tensorrt with rtx 5090s on latest drivers
I followed the steps to run a huggingface model on Tensorrt by using Qwen3-32b, quantizing to fp4 and running.
This was in the examples folder

Expected behavior

loads

actual behavior

hangs

additional notes

no real outputs from tensorrt to help sorry

[Notbici]

NVIDIA/TensorRT#4501 (comment)

[ChiikawaSama]

same bug on official docker image or pre-build wheel, hangs like forever

[Superjomn]

@Notbici Can you export TLLM_LLM_ENABLE_DEBUG=1, it should dump more logs and help investigate.

[darraghdog]

@Superjomn I have this issue also, you can see the logs here and export TLLM_LLM_ENABLE_DEBUG=1 is enabled.
I have opened up another issue for it - #5854.

I have done a lot of testing of different settings, using a kaggle notebook with 2 X T4 GPUs.

• A notebook using tp=1 with trtllm-serve does not work, see here
• A notebook using tp=1 with python -m tensorrt_llm.commands.serve serve works but the same messages Generating a new HMAC key comes up and the model seems to launch twice, see here
• A notebook using tp=2 with python -m tensorrt_llm.commands.serve serve does not work, see here

A snippet of the logs for the second bullet above is shown here. You should be able to copy any of these and run them yourself for free if you create an account.

[07/09/2025-15:11:33] [TRT-LLM] [I] Memory bandwidth: 320 GB/s
[07/09/2025-15:11:33] [TRT-LLM] [I] PCIe speed: 2500 Mbps
[07/09/2025-15:11:33] [TRT-LLM] [I] PCIe link width: 16
[07/09/2025-15:11:33] [TRT-LLM] [I] PCIe bandwidth: 5 GB/s
LLM.args.mpi_session: None
create pool session ...
rank 0 using MpiPoolSession to spawn MPI processes
Server [proxy_request_queue] bound to tcp://127.0.0.1:32921 in PAIR
Server [worker_init_status_queue] bound to tcp://127.0.0.1:44465 in PAIR
Server [proxy_result_queue] bound to tcp://127.0.0.1:45445 in PAIR
Server [proxy_stats_queue] bound to tcp://127.0.0.1:34399 in PAIR
Server [proxy_kv_cache_events_queue] bound to tcp://127.0.0.1:45621 in PAIR
[07/09/2025-15:12:11] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_queue
[07/09/2025-15:12:11] [TRT-LLM] [I] Generating a new HMAC key for server worker_init_status_queue
[07/09/2025-15:12:11] [TRT-LLM] [I] Generating a new HMAC key for server proxy_result_queue
[07/09/2025-15:12:11] [TRT-LLM] [I] Generating a new HMAC key for server proxy_stats_queue
[07/09/2025-15:12:11] [TRT-LLM] [I] Generating a new HMAC key for server proxy_kv_cache_events_queue
...
Multiple distributions found for package optimum. Picked distribution: optimum
LLM debug mode enabled.
[07/09/2025-15:12:54] [TRT-LLM] [I] Starting TensorRT-LLM init.
...
[07/09/2025-15:12:54] [TRT-LLM] [I] TensorRT-LLM inited.
[TensorRT-LLM] TensorRT-LLM version: 1.0.0rc1
Worker 0 entering worker_main...
Worker 0 ready to setup backend...
[07/09/2025-15:13:01] [TRT-LLM] [W] Found worker process 108 was bound to {0, 2}, this may harm performance.
[07/09/2025-15:13:01] [TRT-LLM] [W] Will clear the cpu affinity
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Engine version 1.0.0rc1 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] MPI size: 1, MPI local size: 1, rank: 0
[TensorRT-LLM][INFO] Using user-specified devices: (0)

[rnik12]

Built from source same issue, stuck like forever CPU at 100% only one core used.

>>> llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
rank 0 using MpiPoolSession to spawn MPI processes

[meowmeowxw]

NVIDIA/TensorRT#4501 (comment)

These environment variables solve the hang issue for me as well:

set -euo pipefail
# Keep Open MPI on localhost and avoid fabric auto-detect stalls
export OMPI_MCA_oob=tcp
export OMPI_MCA_oob_tcp_if_include=lo
export OMPI_MCA_oob_tcp_peer_retries=60
export OMPI_MCA_oob_tcp_connect_sleep=10
export OMPI_MCA_btl=self,vader,tcp
export OMPI_MCA_btl_tcp_if_include=lo
export OMPI_MCA_pml=ob1

[karljang]

@Notbici ,
Apologies for the very delayed response, but could you please try the latest version or the above suggestions to see if the issue still remains?

[github-actions]

Issue has not received an update in over 14 days. Adding stale label.