Invalid node error when generating model engine using trtexec for onnx model "ESS DNN stereo disparity" from NGC catalog

[arvindrk2]

I am trying to generate the model engine from the "ESS DNN stereo disparity" onnx file on my jetson orin nano using trtexec (and tensorrt10.3.0). I am using the model version v4.1.0_onnx. But I keep running into the error shown below. Can someone help me understand why I am running into this and how I can fix it?

[06/13/2025-19:41:36] [E] [TRT] parsers/onnx/ModelImporter.cpp:961: While parsing node number 267 [Conv -> "/model/prediction/layers/layers.0/Conv_output_0"]:
[06/13/2025-19:41:36] [E] [TRT] parsers/onnx/ModelImporter.cpp:964: --- Begin node ---
input: "/model/prediction/Reshape_output_0"
input: "model.prediction.layers.0.weight"
input: "model.prediction.layers.0.bias"
output: "/model/prediction/layers/layers.0/Conv_output_0"
name: "/model/prediction/layers/layers.0/Conv"
op_type: "Conv"
attribute {
name: "dilations"
ints: 1
type: INTS
}
attribute {
name: "group"
i: 1
type: INT
}
attribute {
name: "kernel_shape"
ints: 1
type: INTS
}
attribute {
name: "pads"
ints: 0
ints: 0
type: INTS
}
attribute {
name: "strides"
ints: 1
type: INTS
}
[06/13/2025-19:41:36] [E] [TRT] parsers/onnx/ModelImporter.cpp:965: --- End node ---
[06/13/2025-19:41:36] [E] [TRT] parsers/onnx/ModelImporter.cpp:967: ERROR: parsers/onnx/ModelImporter.cpp:202 In function parseNode:
[6] Invalid Node - /model/prediction/layers/layers.0/Conv
[06/13/2025-19:41:36] [E] Failed to parse onnx file
[06/13/2025-19:41:36] [I] Finished parsing network model. Parse time: 0.39815
[06/13/2025-19:41:36] [E] Parsing model failed
[06/13/2025-19:41:36] [E] Failed to create engine from model or file.
[06/13/2025-19:41:36] [E] Engine set up failed

[pranavm-nvidia]

Could you try this with a newer version of TensorRT?

[arvindrk2]

Could you try this with a newer version of TensorRT?

So, when I generate the engine using trtexec on my laptop (WSL2, tensorrt10.11.0), it throws a few "fusedConcatConv3x3" errors but it is still able to get past it and successfully complete the process. However, I need to use the model on my jetson orin nano for my project and I can't use the engine generated on my laptop for this purpose.

[pranavm-nvidia]

I see. I'm not entirely sure what the bug is in 10.3, but my guess would be that it's related to 1D convs. I wonder if you can unsqueeze the input to the Conv and then adjust its attributes to be 2D?

The other option would be to wait for JetPack 7, which is supposed to be available in Q3 (see the roadmap).

[arvindrk2]

Do you know which tensorrt version will be packaged with jetpack 7? I don't mind waiting for Jetpack 7 but I don't know, with certainty, that it will resolve the issue.
I am curious to try and modify this model to help me understand the inside workings a little better. I learnt how to edit the attributes, but if you get the chance, can you please explain exactly what this means "I wonder if you can unsqueeze the input to the Conv and then adjust its attributes to be 2D?" I can then try it out myself.

[pranavm-nvidia]

It's supposed to be TRT 10.11.

The only thing different about this Conv compared to the others in the model is that it only has 1 spatial dimension, while the others have 2. My suspicion is that there was a bug in the ONNX parser related to 1D Convs in older versions of TRT.

Currently, the input tensor is of shape (N, C, H) - if you unsqueeze it to (N, C, H, 1) and then update the attributes, i.e. instead of dilation=1, kernel_shape=1, pads=0, 0, stride=1, you would have dilation=1, 1, kernel_shape=1, 1, pads=0, 0, 0, 0, stride=1, 1, then you can essentially spoof it as a 2D Conv, which might parse correctly.

[arvindrk2]

Thanks for the explanation, @pranavm-nvidia. The modifications you suggested fixed the issue and I was able to generate the engine.

[oUp2Uo]

Hi @arvindrk2 , how do you fix this problem?
I am using TensorRT-10.11.0.33 & TensorRT-10.13.3.9 (win11, CUDA12), and I could not generate the engine too.
Thank you.

[10/16/2025-11:51:21] [I] [TRT] ----------------------------------------------------------------
[10/16/2025-11:51:21] [I] [TRT] Searching for plugin wth node domain namespace:
[10/16/2025-11:51:21] [I] [TRT] Searching for plugin wth node domain namespace:
[10/16/2025-11:51:21] [I] [TRT] Searching for plugin wth node domain namespace:
[10/16/2025-11:51:21] [I] [TRT] Searching for plugin wth node domain namespace:
[10/16/2025-11:51:21] [I] [TRT] Searching for plugin wth node domain namespace:
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:137: While parsing node number 178 [FusedConcatConv3x3 -> "/model/feature_extractor/lastconv/lastconv.3/Conv_output_0"]:
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:138: --- Begin node ---
input: "/model/feature_extractor/layer1/layer1.1/Add_output_0"
input: "/model/feature_extractor/Resize_3_output_0"
input: "/model/feature_extractor/Resize_2_output_0"
input: "/model/feature_extractor/Resize_1_output_0"
input: "/model/feature_extractor/Resize_output_0"
input: "onnx::Conv_713"
input: "onnx::Conv_714"
input: "model.feature_extractor.lastconv.3.weight"
output: "/model/feature_extractor/lastconv/lastconv.3/Conv_output_0"
name: "FusedConcatConv3x3_left"
op_type: "FusedConcatConv3x3"
attribute {
  name: "dilations_Conv1"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group_Conv1"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape_Conv1"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads_Conv1"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides_Conv1"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "dilations_Conv2"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group_Conv2"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape_Conv2"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "pads_Conv2"
  ints: 0
  ints: 0
  ints: 0
  ints: 0
  type: INTS
}
attribute {
  name: "strides_Conv2"
  ints: 1
  ints: 1
  type: INTS
}

[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:139: --- End node ---
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:142: ERROR: onnxOpCheckers.cpp:838 In function checkFallbackPluginImporter:
[6] creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:137: While parsing node number 179 [FusedConcatConv3x3 -> "/model/feature_extractor/lastconv/lastconv.3_1/Conv_output_0"]:
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:138: --- Begin node ---
input: "/model/feature_extractor/layer1/layer1.1_1/Add_output_0"
input: "/model/feature_extractor_1/Resize_3_output_0"
input: "/model/feature_extractor_1/Resize_2_output_0"
input: "/model/feature_extractor_1/Resize_1_output_0"
input: "/model/feature_extractor_1/Resize_output_0"
input: "onnx::Conv_749"
input: "onnx::Conv_750"
input: "model.feature_extractor.lastconv.3.weight"
output: "/model/feature_extractor/lastconv/lastconv.3_1/Conv_output_0"
name: "FusedConcatConv3x3_right"
op_type: "FusedConcatConv3x3"
attribute {
  name: "dilations_Conv1"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group_Conv1"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape_Conv1"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads_Conv1"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides_Conv1"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "dilations_Conv2"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group_Conv2"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape_Conv2"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "pads_Conv2"
  ints: 0
  ints: 0
  ints: 0
  ints: 0
  type: INTS
}
attribute {
  name: "strides_Conv2"
  ints: 1
  ints: 1
  type: INTS
}

[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:139: --- End node ---
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:142: ERROR: onnxOpCheckers.cpp:838 In function checkFallbackPluginImporter:
[6] creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:137: While parsing node number 180 [FusedEinsum2Softmax -> "/model/cost_volume/layers/layers.0/Softmax_output_0"]:
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:138: --- Begin node ---
input: "/model/feature_extractor/lastconv/lastconv.3/Conv_output_0"
input: "/model/feature_extractor/lastconv/lastconv.3_1/Conv_output_0"
output: "/model/cost_volume/layers/layers.0/Softmax_output_0"
name: "FusedEinsum2Softmax"
op_type: "FusedEinsum2Softmax"

[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:139: --- End node ---
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:142: ERROR: onnxOpCheckers.cpp:838 In function checkFallbackPluginImporter:
[6] creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:137: While parsing node number 259 [FusedConcat2c128c64Conv3x3c64 -> "/model/stereo_matching/decoder/blocks.3/blocks.3.1/Relu_output_0"]:
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:138: --- Begin node ---
input: "/model/stereo_matching/encoder/blocks.0/blocks.0.2/Relu_output_0"
input: "/model/stereo_matching/decoder/Resize_3_output_0"
input: "model.stereo_matching.decoder.blocks.3.0.weight"
input: "model.stereo_matching.decoder.blocks.3.0.bias"
output: "/model/stereo_matching/decoder/blocks.3/blocks.3.1/Relu_output_0"
name: "FusedConcat2c128c64Conv3x3c64"
op_type: "FusedConcat2c128c64Conv3x3c64"
attribute {
  name: "dilations_Conv"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group_Conv"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape_Conv"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads_Conv"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides_Conv"
  ints: 1
  ints: 1
  type: INTS
}

[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:139: --- End node ---
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:142: ERROR: onnxOpCheckers.cpp:838 In function checkFallbackPluginImporter:
[6] creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:137: While parsing node number 264 [FusedConcat2c64c64Conv3x3c64 -> "/model/stereo_matching/decoder/blocks.4/blocks.4.1/Relu_output_0"]:
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:138: --- Begin node ---
input: "/model/stereo_matching/encoder/firstconv/firstconv.1/Relu_output_0"
input: "/model/stereo_matching/decoder/Resize_4_output_0"
input: "model.stereo_matching.decoder.blocks.4.0.weight"
input: "model.stereo_matching.decoder.blocks.4.0.bias"
output: "/model/stereo_matching/decoder/blocks.4/blocks.4.1/Relu_output_0"
name: "FusedConcat2c64c64Conv3x3c64"
op_type: "FusedConcat2c64c64Conv3x3c64"
attribute {
  name: "dilations_Conv"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group_Conv"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape_Conv"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads_Conv"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides_Conv"
  ints: 1
  ints: 1
  type: INTS
}

[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:139: --- End node ---
[10/16/2025-11:51:21] [E] [TRT] ModelImporter.cpp:142: ERROR: onnxOpCheckers.cpp:838 In function checkFallbackPluginImporter:
[6] creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[10/16/2025-11:51:21] [E] Failed to parse onnx file
[10/16/2025-11:51:21] [I] Finished parsing network model. Parse time: 0.250286
[10/16/2025-11:51:21] [E] Parsing model failed
[10/16/2025-11:51:21] [E] Failed to create engine from model or file.
[10/16/2025-11:51:21] [E] Engine set up failed

[oUp2Uo]

Hi @pranavm-nvidia , do you have any idea about @arvindrk2 mentioned FusedConcatConv3x3?

After yesterday's test on Win, I tried to run trtexec on Ubuntu today(Jetson Orin NX, jetpack 5.1.1, using TensorRT 8), and add --plugins=ess_plugins.so.
But still the same error message FusedConcatConv3x3.
And engine file could not be generated.

[pranavm-nvidia]

@oUp2Uo My suggestion was to replace 1D convs with 2D convs where one of the dimensions is 1. I'm not sure if that's the same issue you're seeing (and it should have already been fixed in the versions you tested). Are you able to parse this model with other backends, e.g. ONNX-RT? That would be a good sanity check to make sure the model is valid.

[oUp2Uo]

Hi @pranavm-nvidia , thanks for the reply.
I could not inference the model using onnxruntime(1.22.0).
It failed when creating the ort session.
The error thrown is ORT_INVALID_GRAPH.

[pranavm-nvidia]

Seems likely to be an invalid model then. Maybe a bug on the exporter side?