Inconsistency between the outputs of dino onnx and dino trt engine

[Desperado721]

Hi team,

I'm using trtexec to convert dino onnx to trt engine. however I'm seeing different outputs compared to dino onnx. here is what I found: Would appreciate it if you could share your insights. Thanks!

• I tested the outputs on 2 images and visualized the results

• dino torch vs dino onnx: quite similar
• • conf score: almost same
• • bbox: shifted a little bit, but it's okay
• dino onnx vs dino trtr: totally different
• • conf score: is decreased by one order of magnitude.
• • bbox: totally different

Some context:

I'm using mmdeploy to convert dino torch -> dino onnx -> dino trt,

• • I need to build custom ops (gird_sampler ) for onnxruntime and trt separately to support the conversion. Everything looks good to me and I built these dynamic libraries successfully. They are used to complete the conversion and launch triton server for inference. I attached them too for further debug

I used the following cmd to do the conversion, (I'm assuming dino torch->dino onnx succeeded because the outputs are consistent) so I only post the cmd that I used in the 2nd stage(dino onnx -> dino trt)

/azureuser/TensorRT-10.3.0.26/targets/x86_64-linux-gnu/bin/trtexec --onnx=mmdeploy_model/dino_trt_aman_fp16_0304_torchscript/end2end.onnx  --saveEngine=mmdeploy_model/dino_trt_aman_fp16_0304_torchscript/end2end_fp16.plan --minShapes=input:1x3x240x240 --optShapes=input:2x3x240x240 --maxShapes=input:4x3x240x240 --useCudaGraph --plugins=mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so --verbose

dependencies:
Github repo: https://github.com/open-mmlab/mmdeploy/tree/main

My VM:
Cuda 12.6
Standard NC4as T4 v3

Dependencies:
Torch 2.1.0
Onnx 1.19.0
TensorRT 10.3.0
Torchscript https://download.pytorch.org/libtorch/cu121
Triton 24.08

full logs

trtexec.log

my PR for mmdeploy to build custom ops
Desperado721/mmdeploy#1

the virualization of dino onnx, dino torch, dino trt

the dynamic libraries that I built

dynamic_libraries.zip

[lix19937]

How about the trt-fp32 result ? fp16 maybe has some diff .

[2730gf]

@lix19937 Hello, can you help me take a look at this issue? It is also an inconsistency issue.
#4400

[Desperado721]

How about the trt-fp32 result ? fp16 maybe has some diff .

it is trt-fp32, I tried both actually, but neither of them outputs consistent results

[Desperado721]

Hi @lix19937 , Could I catch up a little bit? any insights towards debugging? Thank you

[yuanyao-nv]

For accuracy issues I suggest first comparing onnxruntime inference result with TRT and Torch to see if the issue is in ONNX export or TRT.

[lix19937]

@Desperado721 Try to use ploygraphy to mark all out, to compare ort vs trt .

[Desperado721]

@Desperado721 Try to use ploygraphy to mark all out, to compare ort vs trt .

Thank you for your reply. I used ploygraphy to do a comparison and here what I got

[I] TF32 is disabled by default. Turn on TF32 for better performance with minor accuracy differences.
[I] trt-runner-N0-04/16/25-00:37:05     | Activating and starting inference
[I] Loading plugin library: mmdeploy/build/lib/libmmdeploy_tensorrt_ops.so
[I] Loading plugin library: mmdeploy/build/lib/libmmdeploy_onnxruntime_ops.so
[I] Loading plugin library: mmdeploy/build/lib/libmmdeploy_tensorrt_ops.so
[I] Loading plugin library: mmdeploy/build/lib/libmmdeploy_onnxruntime_ops.so
[I] Loading plugin library: mmdeploy/build/lib/libmmdeploy_tensorrt_ops.so
[I] Loading plugin library: mmdeploy/build/lib/libmmdeploy_onnxruntime_ops.so
[W] ModelImporter.cpp:797: Make sure output labels has Int64 binding.
[I] Configuring with profiles:[
        Profile 0:
            {input [min=[1, 3, 240, 240], opt=[1, 3, 240, 240], max=[1, 3, 240, 240]]}
    ]
[W] profileSharing0806 is on by default in TensorRT 10.0. This flag is deprecated and has no effect.
[I] Building engine with configuration:
    Flags                  | []
    Engine Capability      | EngineCapability.STANDARD
    Memory Pools           | [WORKSPACE: 15935.56 MiB, TACTIC_DRAM: 15935.56 MiB, TACTIC_SHARED_MEMORY: 1024.00 MiB]
    Tactic Sources         | [EDGE_MASK_CONVOLUTIONS, JIT_CONVOLUTIONS]
    Profiling Verbosity    | ProfilingVerbosity.DETAILED
    Preview Features       | [PROFILE_SHARING_0806]
[I] Finished engine building in 94.193 seconds
[I] trt-runner-N0-04/16/25-00:37:05    
    ---- Inference Input(s) ----
    {input [dtype=float32, shape=(1, 3, 240, 240)]}
[I] trt-runner-N0-04/16/25-00:37:05    
    ---- Inference Output(s) ----
    {dets [dtype=float32, shape=(1, 300, 5)],
     labels [dtype=int64, shape=(1, 300)]}
[I] trt-runner-N0-04/16/25-00:37:05     | Completed 1 iteration(s) in 1892 ms | Average inference time: 1892 ms.
[I] onnxrt-runner-N0-04/16/25-00:37:05  | Activating and starting inference
[I] Creating ONNX-Runtime Inference Session with providers: ['CPUExecutionProvider']
[I] onnxrt-runner-N0-04/16/25-00:37:05 
    ---- Inference Input(s) ----
    {input [dtype=float32, shape=(1, 3, 240, 240)]}
[I] onnxrt-runner-N0-04/16/25-00:37:05 
    ---- Inference Output(s) ----
    {dets [dtype=float32, shape=(1, 300, 5)],
     labels [dtype=int64, shape=(1, 300)]}
[I] onnxrt-runner-N0-04/16/25-00:37:05  | Completed 1 iteration(s) in 1.96e+04 ms | Average inference time: 1.96e+04 ms.
[I] Accuracy Comparison | trt-runner-N0-04/16/25-00:37:05 vs. onnxrt-runner-N0-04/16/25-00:37:05
[I]     Comparing Output: 'dets' (dtype=float32, shape=(1, 300, 5)) with 'dets' (dtype=float32, shape=(1, 300, 5))
[I]         Tolerance: [abs=0.0001, rel=0.001] | Checking elemwise error
[W]         onnxrt-runner-N0-04/16/25-00:37:05  | Output: dets: Some values are 0. Will add a small epsilon quantity to these when computing relative difference. Note that this may cause some relative differences to be extremely high. 
[I]         trt-runner-N0-04/16/25-00:37:05: dets | Stats: mean=48.145, std-dev=48.365, var=2339.2, median=30.311, min=0.022697 at (0, 299, 4), max=177.61 at (0, 78, 3), avg-magnitude=48.145, p90=131.31, p95=141.24, p99=155.87
[I]             ---- Histogram ----
                Bin Range  |  Num Elems | Visualization
                (0  , 24 ) |        651 | ########################################
                (24 , 48 ) |        319 | ###################
                (48 , 72 ) |        143 | ########
                (72 , 96 ) |         59 | ###
                (96 , 120) |        105 | ######
                (120, 144) |        159 | #########
                (144, 168) |         60 | ###
                (168, 192) |          4 | 
                (192, 216) |          0 | 
                (216, 240) |          0 | 
[I]         onnxrt-runner-N0-04/16/25-00:37:05: dets | Stats: mean=106.63, std-dev=102.02, var=10407, median=62.029, min=0 at (0, 1, 0), max=240 at (0, 0, 3), avg-magnitude=106.63, p90=239.9, p95=240, p99=240
[I]             ---- Histogram ----
                Bin Range  |  Num Elems | Visualization
                (0  , 24 ) |        531 | ########################################
                (24 , 48 ) |         57 | ####
                (48 , 72 ) |        198 | ##############
                (72 , 96 ) |         56 | ####
                (96 , 120) |         45 | ###
                (120, 144) |         40 | ###
                (144, 168) |         37 | ##
                (168, 192) |         22 | #
                (192, 216) |         19 | #
                (216, 240) |        495 | #####################################
[I]         Error Metrics: dets
[I]             Minimum Required Tolerance: elemwise error | [abs=235.1] OR [rel=7.9875e+17] (requirements may be lower if both abs/rel tolerances are set)
[I]             Absolute Difference | Stats: mean=82.651, std-dev=75.454, var=5693.3, median=68.616, min=0.0046684 at (0, 295, 4), max=235.1 at (0, 182, 2), avg-magnitude=82.651, p90=205.76, p95=218.13, p99=231.11
[I]                 ---- Histogram ----
                    Bin Range       |  Num Elems | Visualization
                    (0.00467, 23.5) |        491 | ########################################
                    (23.5   , 47  ) |        178 | ##############
                    (47     , 70.5) |         89 | #######
                    (70.5   , 94  ) |        117 | #########
                    (94     , 118 ) |        153 | ############
                    (118    , 141 ) |        120 | #########
                    (141    , 165 ) |         54 | ####
                    (165    , 188 ) |         70 | #####
                    (188    , 212 ) |        116 | #########
                    (212    , 235 ) |        112 | #########
[I]             Relative Difference | Stats: mean=1.1819e+16, std-dev=7.1035e+16, var=5.046e+33, median=0.68947, min=0.004235 at (0, 156, 1), max=7.9875e+17 at (0, 78, 1), avg-magnitude=1.1819e+16, p90=37.627, p95=2617.5, p99=5.2558e+17
[I]                 ---- Histogram ----
                    Bin Range            |  Num Elems | Visualization
                    (0.00424 , 7.99e+16) |       1444 | ########################################
                    (7.99e+16, 1.6e+17 ) |         20 | 
                    (1.6e+17 , 2.4e+17 ) |         13 | 
                    (2.4e+17 , 3.2e+17 ) |          5 | 
                    (3.2e+17 , 3.99e+17) |          0 | 
                    (3.99e+17, 4.79e+17) |          0 | 
                    (4.79e+17, 5.59e+17) |          7 | 
                    (5.59e+17, 6.39e+17) |          7 | 
                    (6.39e+17, 7.19e+17) |          1 | 
                    (7.19e+17, 7.99e+17) |          3 | 
[E]         FAILED | Output: 'dets' | Difference exceeds tolerance (rel=0.001, abs=0.0001)
[I] Output Validation | Runners: ['trt-runner-N0-04/16/25-00:37:05', 'onnxrt-runner-N0-04/16/25-00:37:05']
[I]     trt-runner-N0-04/16/25-00:37:05     | Validating output: dets (check_inf=True, check_nan=True)
[I]         mean=48.145, std-dev=48.365, var=2339.2, median=30.311, min=0.022697 at (0, 299, 4), max=177.61 at (0, 78, 3), avg-magnitude=48.145, p90=131.31, p95=141.24, p99=155.87
[I]         PASSED | Output: dets is valid
[I]     trt-runner-N0-04/16/25-00:37:05     | Validating output: labels (check_inf=True, check_nan=True)
[I]         mean=0.88333, std-dev=1.1789, var=1.3897, median=0, min=0 at (0, 1), max=5 at (0, 194), avg-magnitude=0.88333, p90=2, p95=2.05, p99=4
[I]         PASSED | Output: labels is valid
[I]     onnxrt-runner-N0-04/16/25-00:37:05  | Validating output: dets (check_inf=True, check_nan=True)
[I]         mean=106.63, std-dev=102.02, var=10407, median=62.029, min=0 at (0, 1, 0), max=240 at (0, 0, 3), avg-magnitude=106.63, p90=239.9, p95=240, p99=240
[I]         PASSED | Output: dets is valid
[I]     onnxrt-runner-N0-04/16/25-00:37:05  | Validating output: labels (check_inf=True, check_nan=True)
[I]         mean=1.6367, std-dev=1.3897, var=1.9313, median=2, min=0 at (0, 0), max=4 at (0, 26), avg-magnitude=1.6367, p90=4, p95=4, p99=4
[I]         PASSED | Output: labels is valid
[I]     PASSED | Output Validation
[E] FAILED | Runtime: 122.881s | Command: /usr/local/bin/polygraphy run mmdeploy_model/dino_onnx_aman_0319/end2end.onnx --trt --onnxrt --fail-fast --plugins mmdeploy/build/lib/libmmdeploy_tensorrt_ops.so mmdeploy/build/lib/libmmdeploy_onnxruntime_ops.so --validate --atol 1e-4 --rtol 1e-3 --input-shapes input:[1,3,240,240]

it seems that the inference failed at a very stage because some results are 0. Because I used a custom ops and I got warnings when I built these ops. Does IPluginV2Ext still work in trt 10.x? or do I have to overwrite some functions using its alternative? (IPluginV3)? IPluginV2Ext is marked as deprecated in trt 10.x, but not sure if it is till working. Thanks!

/azureuser/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/common/trt_plugin_base.hpp:11:22: warning: 'IPluginV2Ext' is deprecated [-Wdeprecated-declarations]
   11 | #define TRT_NOEXCEPT noexcept
      |                      ^~~~~~~~
/azureuser/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/batched_rotated_nms/trt_batched_rotated_nms.cpp:180:56: note: in expansion of macro 'TRT_NOEXCEPT'
  180 |     const char* name, const PluginFieldCollection* fc) TRT_NOEXCEPT {
      |                                                        ^~~~~~~~~~~~
In file included from /azureuser/TensorRT-10.3.0.26/include/NvInferRuntimeCommon.h:34,
                 from /azureuser/TensorRT-10.3.0.26/include/NvInferPluginUtils.h:21,
                 from /azureuser/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/batched_rotated_nms/trt_batched_rotated_nms.hpp:8,
                 from /azureuser/mmdeploy/csrc/mmdeploy/backend_ops/tensorrt/batched_rotated_nms/trt_batched_rotated_nms.cpp:3:
/azureuser/TensorRT-10.3.0.26/include/NvInferRuntimePlugin.h:463:22: note: declared here
  463 | class TRT_DEPRECATED IPluginV2Ext : public IPluginV2

[LightSun]

I met a problem , very like this. I tested on two machine. One is cuda11.4 + Trt10.1, another is cuda11.8 + Trt10.1. but the infer result is quite difference. I ensure the infer code is the same.
The first result is

box: (left,top,w,h) = (153, 177, 209, 144), prob = 0.843, txt = 右肾长轴切面
box: (left,top,w,h) = (59, 133, 228, 210), prob = 0.112, txt = 肝右肾矢状切面
box: (left,top,w,h) = (53, 134, 233, 219), prob = 0.382, txt = 肝右肾矢状切面
box: (left,top,w,h) = (103, 137, 182, 168), prob = 0.165, txt = 肝右肾矢状切面
box: (left,top,w,h) = (122, 137, 163, 167), prob = 0.117, txt = 肝右肾矢状切面
box: (left,top,w,h) = (101, 139, 179, 160), prob = 0.117, txt = 肝右肾矢状切面
box: (left,top,w,h) = (120, 137, 163, 159), prob = 0.106, txt = 肝右肾矢状切面

The second result is

box: (left,top,w,h) = (153, 176, 209, 144), prob = 0.815, txt = 右肾长轴切面
box: (left,top,w,h) = (102, 130, 191, 172), prob = 0.120, txt = 肝右肾矢状切面
box: (left,top,w,h) = (60, 127, 233, 211), prob = 0.104, txt = 肝右肾矢状切面
box: (left,top,w,h) = (53, 127, 242, 224), prob = 0.214, txt = 肝右肾矢状切面
box: (left,top,w,h) = (52, 129, 235, 223), prob = 0.219, txt = 肝右肾矢状切面
box: (left,top,w,h) = (49, 126, 248, 230), prob = 0.176, txt = 肝右肾矢状切面

convert cmd of trtexec:

${DRT_DIR}/trtexec --onnx=${MODULE_DIR}/grounded_opset17_dynamic_4.onnx ${FMT} ${WORKSPACE2} --saveEngine=dino_demo_len_static_378.trt

I didn't know why.

[ronghuaiyang]

I met a problem , very like this. I tested on two machine. One is cuda11.4 + Trt10.1, another is cuda11.8 + Trt10.1. but the infer result is quite difference. I ensure the infer code is the same. The first result is

box: (left,top,w,h) = (153, 177, 209, 144), prob = 0.843, txt = 右肾长轴切面
box: (left,top,w,h) = (59, 133, 228, 210), prob = 0.112, txt = 肝右肾矢状切面
box: (left,top,w,h) = (53, 134, 233, 219), prob = 0.382, txt = 肝右肾矢状切面
box: (left,top,w,h) = (103, 137, 182, 168), prob = 0.165, txt = 肝右肾矢状切面
box: (left,top,w,h) = (122, 137, 163, 167), prob = 0.117, txt = 肝右肾矢状切面
box: (left,top,w,h) = (101, 139, 179, 160), prob = 0.117, txt = 肝右肾矢状切面
box: (left,top,w,h) = (120, 137, 163, 159), prob = 0.106, txt = 肝右肾矢状切面

The second result is

box: (left,top,w,h) = (153, 176, 209, 144), prob = 0.815, txt = 右肾长轴切面
box: (left,top,w,h) = (102, 130, 191, 172), prob = 0.120, txt = 肝右肾矢状切面
box: (left,top,w,h) = (60, 127, 233, 211), prob = 0.104, txt = 肝右肾矢状切面
box: (left,top,w,h) = (53, 127, 242, 224), prob = 0.214, txt = 肝右肾矢状切面
box: (left,top,w,h) = (52, 129, 235, 223), prob = 0.219, txt = 肝右肾矢状切面
box: (left,top,w,h) = (49, 126, 248, 230), prob = 0.176, txt = 肝右肾矢状切面

convert cmd of trtexec:

${DRT_DIR}/trtexec --onnx=${MODULE_DIR}/grounded_opset17_dynamic_4.onnx ${FMT} ${WORKSPACE2} --saveEngine=dino_demo_len_static_378.trt

I didn't know why.

have you solve the problem? I met the same.