onnx to engine failure of TensorRT 10.8.0 when running trtexec.exe on GPU 3060TI #4658
Description
Description
(BasicEnv) PS E:\Development\1.AutoTrace\2.Model\Pytorch-UNet\saved_models\test> trtexec --onnx=DPUNet_second_argmax_12_quan.onnx --saveEngine=fp32.engine
&&&& RUNNING TensorRT.trtexec [TensorRT v100800] [b43] # C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT\TensorRT-10.8.0.43\bin\trtexec.exe --onnx=DPUNet_second_argmax_12_quan.onnx --saveEngine=fp32.engine
[12/04/2025-16:45:32] [I] TF32 is enabled by default. Add --noTF32 flag to further improve accuracy with some performance cost.
[12/04/2025-16:45:32] [I] === Model Options ===
[12/04/2025-16:45:32] [I] Format: ONNX
[12/04/2025-16:45:32] [I] Model: DPUNet_second_argmax_12_quan.onnx
[12/04/2025-16:45:32] [I] Output:
[12/04/2025-16:45:32] [I] === Build Options ===
[12/04/2025-16:45:32] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default, tacticSharedMem: default
[12/04/2025-16:45:32] [I] avgTiming: 8
[12/04/2025-16:45:32] [I] Precision: FP32
[12/04/2025-16:45:32] [I] LayerPrecisions:
[12/04/2025-16:45:32] [I] Layer Device Types:
[12/04/2025-16:45:32] [I] Calibration:
[12/04/2025-16:45:32] [I] Refit: Disabled
[12/04/2025-16:45:32] [I] Strip weights: Disabled
[12/04/2025-16:45:32] [I] Version Compatible: Disabled
[12/04/2025-16:45:32] [I] ONNX Plugin InstanceNorm: Disabled
[12/04/2025-16:45:32] [I] TensorRT runtime: full
[12/04/2025-16:45:32] [I] Lean DLL Path:
[12/04/2025-16:45:32] [I] Tempfile Controls: { in_memory: allow, temporary: allow }
[12/04/2025-16:45:32] [I] Exclude Lean Runtime: Disabled
[12/04/2025-16:45:32] [I] Sparsity: Disabled
[12/04/2025-16:45:32] [I] Safe mode: Disabled
[12/04/2025-16:45:32] [I] Build DLA standalone loadable: Disabled
[12/04/2025-16:45:32] [I] Allow GPU fallback for DLA: Disabled
[12/04/2025-16:45:32] [I] DirectIO mode: Disabled
[12/04/2025-16:45:32] [I] Restricted mode: Disabled
[12/04/2025-16:45:32] [I] Skip inference: Disabled
[12/04/2025-16:45:32] [I] Save engine: fp32.engine
[12/04/2025-16:45:32] [I] Load engine:
[12/04/2025-16:45:32] [I] Profiling verbosity: 0
[12/04/2025-16:45:32] [I] Tactic sources: Using default tactic sources
[12/04/2025-16:45:32] [I] timingCacheMode: local
[12/04/2025-16:45:32] [I] timingCacheFile:
[12/04/2025-16:45:32] [I] Enable Compilation Cache: Enabled
[12/04/2025-16:45:32] [I] Enable Monitor Memory: Disabled
[12/04/2025-16:45:32] [I] errorOnTimingCacheMiss: Disabled
[12/04/2025-16:45:32] [I] Preview Features: Use default preview flags.
[12/04/2025-16:45:32] [I] MaxAuxStreams: -1
[12/04/2025-16:45:32] [I] BuilderOptimizationLevel: -1
[12/04/2025-16:45:32] [I] MaxTactics: -1
[12/04/2025-16:45:32] [I] Calibration Profile Index: 0
[12/04/2025-16:45:32] [I] Weight Streaming: Disabled
[12/04/2025-16:45:32] [I] Runtime Platform: Same As Build
[12/04/2025-16:45:32] [I] Debug Tensors:
[12/04/2025-16:45:32] [I] Input(s)s format: fp32:CHW
[12/04/2025-16:45:32] [I] Output(s)s format: fp32:CHW
[12/04/2025-16:45:32] [I] Input build shapes: model
[12/04/2025-16:45:32] [I] Input calibration shapes: model
[12/04/2025-16:45:32] [I] === System Options ===
[12/04/2025-16:45:32] [I] Device: 0
[12/04/2025-16:45:32] [I] DLACore:
[12/04/2025-16:45:32] [I] Plugins:
[12/04/2025-16:45:32] [I] setPluginsToSerialize:
[12/04/2025-16:45:32] [I] dynamicPlugins:
[12/04/2025-16:45:32] [I] ignoreParsedPluginLibs: 0
[12/04/2025-16:45:32] [I]
[12/04/2025-16:45:32] [I] === Inference Options ===
[12/04/2025-16:45:32] [I] Batch: Explicit
[12/04/2025-16:45:32] [I] Input inference shapes: model
[12/04/2025-16:45:32] [I] Iterations: 10
[12/04/2025-16:45:32] [I] Duration: 3s (+ 200ms warm up)
[12/04/2025-16:45:32] [I] Sleep time: 0ms
[12/04/2025-16:45:32] [I] Idle time: 0ms
[12/04/2025-16:45:32] [I] Inference Streams: 1
[12/04/2025-16:45:32] [I] ExposeDMA: Disabled
[12/04/2025-16:45:32] [I] Data transfers: Enabled
[12/04/2025-16:45:32] [I] Spin-wait: Disabled
[12/04/2025-16:45:32] [I] Multithreading: Disabled
[12/04/2025-16:45:32] [I] CUDA Graph: Disabled
[12/04/2025-16:45:32] [I] Separate profiling: Disabled
[12/04/2025-16:45:32] [I] Time Deserialize: Disabled
[12/04/2025-16:45:32] [I] Time Refit: Disabled
[12/04/2025-16:45:32] [I] NVTX verbosity: 0
[12/04/2025-16:45:32] [I] Persistent Cache Ratio: 0
[12/04/2025-16:45:32] [I] Optimization Profile Index: 0
[12/04/2025-16:45:32] [I] Weight Streaming Budget: 100.000000%
[12/04/2025-16:45:32] [I] Inputs:
[12/04/2025-16:45:32] [I] Debug Tensor Save Destinations:
[12/04/2025-16:45:32] [I] === Reporting Options ===
[12/04/2025-16:45:32] [I] Verbose: Disabled
[12/04/2025-16:45:32] [I] Averages: 10 inferences
[12/04/2025-16:45:32] [I] Percentiles: 90,95,99
[12/04/2025-16:45:32] [I] Dump refittable layers:Disabled
[12/04/2025-16:45:32] [I] Dump output: Disabled
[12/04/2025-16:45:32] [I] Profile: Disabled
[12/04/2025-16:45:32] [I] Export timing to JSON file:
[12/04/2025-16:45:32] [I] Export output to JSON file:
[12/04/2025-16:45:32] [I] Export profile to JSON file:
[12/04/2025-16:45:32] [I]
[12/04/2025-16:45:32] [I] === Device Information ===
[12/04/2025-16:45:32] [I] Available Devices:
[12/04/2025-16:45:32] [I] Device 0: "NVIDIA GeForce RTX 3060 Ti" UUID: GPU-9c6a7d89-1783-8e5c-80dd-f4e704947787
[12/04/2025-16:45:32] [I] Selected Device: NVIDIA GeForce RTX 3060 Ti
[12/04/2025-16:45:32] [I] Selected Device ID: 0
[12/04/2025-16:45:32] [I] Selected Device UUID: GPU-9c6a7d89-1783-8e5c-80dd-f4e704947787
[12/04/2025-16:45:32] [I] Compute Capability: 8.6
[12/04/2025-16:45:32] [I] SMs: 38
[12/04/2025-16:45:32] [I] Device Global Memory: 8191 MiB
[12/04/2025-16:45:32] [I] Shared Memory per SM: 100 KiB
[12/04/2025-16:45:32] [I] Memory Bus Width: 256 bits (ECC disabled)
[12/04/2025-16:45:32] [I] Application Compute Clock Rate: 1.665 GHz
[12/04/2025-16:45:32] [I] Application Memory Clock Rate: 7.001 GHz
[12/04/2025-16:45:32] [I]
[12/04/2025-16:45:32] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[12/04/2025-16:45:32] [I]
[12/04/2025-16:45:32] [I] TensorRT version: 10.8.0
[12/04/2025-16:45:32] [I] Loading standard plugins
[12/04/2025-16:45:33] [I] [TRT] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 22427, GPU 1063 (MiB)
[12/04/2025-16:45:36] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +3829, GPU +412, now: CPU 26601, GPU 1475 (MiB)
[12/04/2025-16:45:36] [I] Start parsing network model.
[12/04/2025-16:45:36] [I] [TRT] ----------------------------------------------------------------
[12/04/2025-16:45:36] [I] [TRT] Input filename: DPUNet_second_argmax_12_quan.onnx
[12/04/2025-16:45:36] [I] [TRT] ONNX IR version: 0.0.8
[12/04/2025-16:45:36] [I] [TRT] Opset version: 17
[12/04/2025-16:45:36] [I] [TRT] Producer name: pytorch
[12/04/2025-16:45:36] [I] [TRT] Producer version: 2.5.1
[12/04/2025-16:45:36] [I] [TRT] Domain:
[12/04/2025-16:45:36] [I] [TRT] Model version: 0
[12/04/2025-16:45:36] [I] [TRT] Doc string:
[12/04/2025-16:45:36] [I] [TRT] ----------------------------------------------------------------
[12/04/2025-16:45:36] [I] Finished parsing network model. Parse time: 0.187905
[12/04/2025-16:45:37] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[12/04/2025-16:46:25] [E] Error[9]: Error Code: 9: Skipping tactic 0xa4ae2d82115c3e83 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xff6944b17d5b2e32 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x69c4e2ca38eadce2 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xfe80445f7bb61a99 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x3f2b14dec582741e due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xb5f710b331ba8377 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x9ec201b34455146e due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x2468d082d0ff7c9a due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xe742f4598442d2f1 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xb86b920539eb9c79 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x91930a570b557437 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xcd229658c16b33cd due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x5f1a472d416ff35e due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x65fbe45b4cb1d8a5 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xdc1f355deb032b87 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xa2f15b0a75b7dcec due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x6986b0c6a136276e due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x8141573686849b61 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x844ea9c00f711f19 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xa71946688cad8664 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x5bd8221bd57baf93 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x33a5c6dd086942c1 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x60b880e28fee7a0c due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xcf64a2ae51bf6b36 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xbeb5d91e1874a437 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xdf7e1bd6a496d667 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x445983715412fbda due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x1e55f8b415964e81 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x9003f5f7ff9b1aec due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x991db9fd58152c33 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x84942841d92b0552 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x748f926574b7bdc4 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x271b998fe31732ef due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x7720f198395e7d3d due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x6e9c17a33c93d9b0 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x960e9baa2a6cad5b due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x6d377e4222886190 due to exception Assertion vol == 1 failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x37fdafc098cbcbca due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x2ad0b2590a0074d9 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x82e49cb65905fd80 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x6d976045f181cb7d due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x3c4f10ffbef02293 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x270cd4a3ab8d2a17 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xe4e3d579c41261e4 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x42a23284d8f6bbb0 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x6f78c934a07fe397 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x5aec8106e3de504d due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xe4f5bce9b8ea934a due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x78882e2a485fb681 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x1e837935c924adb5 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x9aaaf14d64bd6d88 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x9549ef6b9c32e5df due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x4c8f386f095b440e due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xfe6dbef234ecea30 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x82b908757412b0c9 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x722ba4df150451c0 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x2f8257256c6b68c8 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xfca95222667b2418 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x2cbe6b9c8db6c34e due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x57ccfc331c264c8a due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x473d87502f60ad57 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xb95c4f13c185aa42 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x259b57af811f5f57 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xe97f57cfcc437eea due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xc70302b71dd25480 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0xa100a1d81f141d10 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[9]: Error Code: 9: Skipping tactic 0x5c7e430c3a1da5d3 due to exception Assertion vol == GK failed.
[12/04/2025-16:47:17] [E] Error[10]: IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node dec_layers.0.upsample.conv_trans.weight + /upsample/conv_trans/_weight_quantizer/QuantizeLinear + /upsample/conv_trans/ConvTranspose.)
[12/04/2025-16:47:17] [E] Engine could not be created from network
[12/04/2025-16:47:17] [E] Building engine failed
[12/04/2025-16:47:17] [E] Failed to create engine from model or file.
[12/04/2025-16:47:17] [E] Engine set up failed
Environment
TensorRT Version: 10.8.0
NVIDIA GPU: 3060TI
NVIDIA Driver Version:
CUDA Version: 12.8
CUDNN Version: 8.6.5
Operating System: Windows 11
Python Version (if applicable): 3.10.18
Tensorflow Version (if applicable): none
PyTorch Version (if applicable): 2.5.1
Baremetal or Container (if so, version): none
Relevant Files
Model link:
Steps To Reproduce
Commands or scripts:
Have you tried the latest release?:
Attach the captured .json and .bin files from TensorRT's API Capture tool if you're on an x86_64 Unix system
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):