wrong results of TensorRT 10.0 when running on GPU Tesla T4

[yflv-yanxia]

Description

The output of the TensorRT 10 model converted from ONNX is incorrect, while the output of the TensorRT 8.6 model is correct. The issue seems to be located in some fully connected layers in the TensorRT 10 model, where the error in the output suddenly becomes very large. The exact cause is unknown. Please help to resolve this issue.

Environment

TensorRT Version: TensorRT 10.0.1

NVIDIA GPU: Tesla T4

NVIDIA Driver Version: 450.36.06

CUDA Version: 11.0

CUDNN Version:8.0.0

Operating System:

onnx opset17

Relevant Files

Model link: https://drive.google.com/file/d/1QBbmtdaecWAHzqMdh10QVbdSjTWzleqo/view?usp=sharing

Steps To Reproduce

1. Convert the ONNX model to TensorRT 10 using ./trtexec --onnx=./test.onnx --device=0 --saveEngine=./test.trtmodel --precisionConstraints=obey.

[lix19937]

Run follow ,then upload the log.

 ./trtexec --onnx=./test.onnx --device=0 --saveEngine=./test.trtmodel   --verbose  2>&1 |tee  build.log

[yflv-yanxia]

build.log
This is the log
@lix19937

[lix19937]

From your log,

[07/15/2024-08:14:59] [E] Error[1]: [cudaResources.cpp::haveStreamOrderedAllocatorHelper::15] Error Code 1: Cuda Runtime (invalid argument)

maybe has problem, also just a warning, need check.
Use follow cmd to compare the result with onnxruntime

polygraphy run test.onnx --trt --onnxrt   

[yflv-yanxia]

log_testonnx.txt
Here are the results after running the above instructions. @lix19937

[lix19937]

The result has a big diff.

Use follow cmd @ trt8.6 and trt10.0 ,and then upload the two li.json

trtexec --onnx=model_sim.onnx --verbose   \
--dumpProfile --dumpLayerInfo --separateProfileRun \
--noDataTransfers --useCudaGraph --useSpinWait --profilingVerbosity=detailed  --exportLayerInfo=li.json

[yflv-yanxia]

li_10.json
li_86.json
Here are the results. @lix19937

[lix19937]

They choose some different tactic, btw two build log can provide ? They run at the same machine ?

[lix19937]

Another, you can try the TensorRT v10.2 .

[yflv-yanxia]

build.log This is the log @lix19937

The build log for TRT10 has been provided before.
Below is the build log for TRT8.6.
build_86.log
The previously provided li_86.json was not obtained on the same machine. I have now obtained li_86.json on the same machine.
li_86(1).json
@lix19937

[lix19937]

You can try to use follow cmd

 polygraphy run model_sim.onnx --trt --onnxrt \
     --trt-outputs mark all \
     --onnx-outputs mark all

[yflv-yanxia]

log_eff.txt
Here are the results. @lix19937

[yflv-yanxia]

Hi, sorry to bother you, but is there any update on the solution? @lix19937

[yflv-yanxia]

I tried TensorRT 10.3, and it's the same error. If it's an issue with the elemwise layer, is there a plan to fix it?

[yflv-yanxia]

Do you have any suggestions for solving this problem? @lix19937

[lix19937]

@yflv-yanxia
Are you compare your onnx result with torch forward ?

If onnx forward result is right, then you can try to add --noTF32 in trtexec cmd like follow:
trtexec --noTF32

From you log, can find /image_encoder/backbone/stages.3/op_list.5/main/inverted_conv/act/Mul_1_output_0 out first raise max diff, you can split your model to check .

[I]         Error Metrics: /image_encoder/backbone/stages.3/op_list.5/main/inverted_conv/act/Mul_1_output_0
[I]             Minimum Required Tolerance: elemwise error | [abs=141.75] OR [rel=3.9318e+06] (requirements may be lower if both abs/rel tolerances are set)
[I]             Absolute Difference | Stats: mean=0.00049691, std-dev=0.07922, var=0.0062759, median=0.00014859, min=0 at (0, 3, 50, 37), max=141.75 at (0, 459, 55, 55), avg-magnitude=0.00049691
[I]                 ---- Histogram ----
                    Bin Range    |  Num Elems | Visualization
                    (0   , 14.2) |    3211263 | ########################################
                    (14.2, 28.4) |          0 | 
                    (28.4, 42.5) |          0 | 
                    (42.5, 56.7) |          0 | 
                    (56.7, 70.9) |          0 | 
                    (70.9, 85.1) |          0 | 
                    (85.1, 99.2) |          0 | 
                    (99.2, 113 ) |          0 | 
                    (113 , 128 ) |          0 | 
                    (128 , 142 ) |          1 |