Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[onnx::Gather_543...Transpose_3934 + Reshape_3943]}.)

[david-PHR]

Description

I use trtexec (TensorRT 8.5.0 from the nvidia NGC pytorch 22.09-py3 container) to convert an ONNX model to trt engine.

There is an issue when I compile the ONNX model. After several minute on a A100-40 GB GPU it show the error below :
[02/07/2023-11:12:18] [W] [TRT] Using kFASTER_DYNAMIC_SHAPES_0805 preview feature. [02/07/2023-11:13:44] [W] [TRT] Skipping tactic 0x0000000000000000 due to Myelin error: Autotuner: no tactics to implement operation [02/07/2023-11:13:44] [E] Error[10]: [optimizer.cpp::computeCosts::3712] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[onnx::Gather_543...Transpose_3934 + Reshape_3943]}.) [02/07/2023-11:13:44] [E] Error[2]: [builder.cpp::buildSerializedNetwork::738] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
Here is the command that I've used:

trtexec --onnx=/home/my_model.onnx --saveEngine=/home/my_model.onnx_trt --fp16 --inputIOFormats=fp16:chw --outputIOFormats=fp16:chw --workspace=40000 --minShapes=input_0:1x3x896x896 --optShapes=input_0:1x3x896x896 --maxShapes=input_0:1x3x896x896 --preview=+fasterDynamicShapes0805

I've tried with smaller workspace size, with and without the fasterDynamicShapes0805 preview features, and with/without dynamic inputs size. Do you have any idea on how to solve this issue?

[david-PHR]

I can now compile the model when updating to TensorRT 8.5.2 for a higher resolution than 512x512.
Unfortunately, I still cannot support dynamic input shape. Let's say that I want to compile the model for 512 to 896 width and height range, this issue appears:

[02/07/2023-12:23:38] [W] [TRT] Using kFASTER_DYNAMIC_SHAPES_0805 preview feature. [02/07/2023-12:24:38] [W] [TRT] Skipping tactic 0x0000000000000000 due to exception Autotuner: no tactics to implement operation [02/07/2023-12:24:38] [E] Error[10]: [optimizer.cpp::computeCosts::3728] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[onnx::Gather_543...Transpose_3934 + Reshape_3943]}.) [02/07/2023-12:24:38] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )

[zerollzeng]

Can you try our latest container first? if it still fails could you please share the onnx with us? We can do further debug. Thanks!

[david-PHR]

I've tried the latest container, which works with static input shapes.
Unfortunately, it still doesn't work when input shapes are dynamic.
I've identified the part of the graph that failed.
Here is the ONNX file of the subgraph that fail:
subgraph_fail.onnx.zip

Here is the command that failed:
trtexec --onnx=subgraph_fail.onnx --saveEngine=subgraph_fail.onnx_trt --fp16 --inputIOFormats=fp16:chw --outputIOFormats=fp16:chw --workspace=16384 --preview=+fasterDynamicShapes0805 --minShapes=input_0:1x3x384x384 --optShapes=input_0:1x3x512x512 --maxShapes=input_0:1x3x896x896

[david-PHR]

Hello, do you have any news on this error?

[zerollzeng]

I can reproduce the error and I've filed internal bug 3979250 for this. Will update here when we have progress on this.

[oxana-nvidia]

bug is fixed on TensorRT 8.6. Please verify when it is publicly available.

&&&& PASSED TensorRT.trtexec [TensorRT v8600] # trtexec --onnx=subgraph_fail.onnx --saveEngine=subgraph_fail.onnx_trt --preview=+fasterDynamicShapes0805 --minShapes=input_0:1x3x384x384 --optShapes=input_0:1x3x512x512 --maxShapes=input_0:1x3x896x896 --verbose

[zerollzeng]

Closed. feel free to reopen if you have any further questions. Thanks!

[FrankyTang]

I have the same error in TensorRT8601.I tried to interrupt this Myelin, but I'm not sure which operators were fused due to the insufficient information provided by myelin，how can I debug 。
[E] Error[10]: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/agent_encoder/Expand_2.../Slice_3]}.)

[oxana-nvidia]

Please try TensorRT 10.0

[JohnHerry]

Same issure on both Tensorrt V9.2 and Tensorrt V10.0.1.6

[E] Error[10]: Could not find any implementation for node {ForeignNode[(Unnamed Layer* 1183)[Cast].../Transpose_1 + ONNXTRT_unsqueezeTensor_12483]}