ONNX to TensorRT taking too long on TensorRT 10.13.2.6 when running on RTX4060 Mobile with CUDA 12.8

[hllllkkkkkk]

Description

Hi, I have an ONNX model and currently I get 30FPS of inferenece on RTX4060 Mobile. I am trying to gain some performance throgh using TensorRT. This is the summary of my ONNX model:

nodes = 2049 initializers = 444 inputs = 2 outputs = 1 top ops: [(‘Constant’, 592), (‘Unsqueeze’, 227), (‘Add’, 163), (‘Transpose’, 133), (‘MatMul’, 132), (‘Concat’, 113), (‘Shape’, 107), (‘Gather’, 103), (‘Reshape’, 95), (‘Mul’, 94), (‘Div’, 53), (‘LayerNormalization’, 48), (‘Conv’, 45), (‘Erf’, 26), (‘Cast’, 26), (‘Slice’, 20), (‘Softmax’, 14), (‘BatchNormalization’, 9), (‘ReduceMean’, 8), (‘Relu’, 7)]

also this is link to onnx file. Now this is where problems start, using this command:

trtexec --onnx=asymformer_160.onnx ^–saveEngine=test.engine ^–fp16 --noTF32 ^–minShapes=img:1x3x160x160,dep:1x1x160x160 ^–optShapes=img:1x3x160x160,dep:1x1x160x160 ^–maxShapes=img:1x3x160x160,dep:1x1x160x160 ^–precisionConstraints=prefer ^–memPoolSize=workspace:2048 ^–tacticSources=+CUBLAS,+CUBLAS_LT

I cannot get it complete it after 10 hours of waiting and gave up.

TensorRT Version: TensorRT-10.13.2.6.Windows.win10.cuda-12.9
GPU Type: RTX4060
Nvidia Driver Version: 576.88
CUDA Version: 12.8
CUDNN Version: cudnn-windows-x86_64-8.9.7.29_cuda12-archive
Operating System + Version: Windows 10
Python Version (if applicable): 3.11.9
TensorFlow Version (if applicable): -
PyTorch Version (if applicable): pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu128

Steps To Reproduce

Install the ONNX file I shared.

1. Run the trtexec command.
2. I actviated verbose flag to see if I can catch any errors, but seemingly everything works but they are just too slow! I don’t know if this is normal or not since I am not familiar with TensorRT.
   Note: This is a copy of this issue from Nvidia forum.

[kevinch-nv]

Some options to try:

• remove -–tacticSources=+CUBLAS,+CUBLAS_LT since it includes more kernels that increases the optimization time for TensorRT
• add --builderOptimizationLevel=0 to have TensorRT choose its tactics faster

[hllllkkkkkk]

I'll try and will turn back to you.

[geraldstanje1]

where to add --builderOptimizationLevel=0 ?