Cuda bindings mismatch error using custom trained model

[immaixq]

Description:
Encounter error using yolov8_tensor_rt.launch.py and isaac_ros_yolov8_visualize.launch.py. The scripts are run on jetson orin nano. The custom trained model is exported to onnx as suggested by documentation using ultralytics, I have also tried exporting using trtexec but still face the same error.

Thank you for your help.

[mzahana]

Hi @immaixq
Have you figured this out?

I am also trying to use a custom yolov8 model, but failed to do so. I also converted the .pt file to onnx and the to an engine file, but I am not sure If I need to change any of the arguments that are defined in the following command

   ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=/tmp/yolov8s.onnx engine_file_path:=/tmp/yolov8s.plan input_binding_names:=['images'] output_binding_names:=['output0'] network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45

[mzahana]

I suspect that the issue is in the hardcoded number of classes

isaac_ros_object_detection/isaac_ros_yolov8/src/yolov8_decoder_node.cpp

Line 76 in b43ecae

int num_classes = 80;

If your custom model has different number of classes then it will break here

isaac_ros_object_detection/isaac_ros_yolov8/src/yolov8_decoder_node.cpp

Line 92 in b43ecae

for (int j = 0; j < num_classes; j++) {

I suggest that you visualize your ONNX model using netron, and note the input/output names and the input/output dimensions.

[kajananchinniahNV]

Hi @immaixq,

The problem you posted seems to originate from a parameter mismatch. We have parameters called input_binding_names and output_binding_names, as well as input_tensor_names and output_tensor_names. Please see here for where they are located in the launch file. I believe that these need to be updated accordingly to match what your model expects.

The input/output binding names should be noted from the ONNX model as @mzahana said. The input tensor names can be kept the same. The output tensor name I believe should be output_tensor based on here.

It also appears that your model outputs a 3 tensors, instead of 2 as we expect. Having additional information about your network would likely be useful.

[kajananchinniahNV]

Hi @mzahana,

To respond to your issue separately, hopefully my first response is helpful for your first issue.

You're also right that the number of classes expected by the decoder node is hard coded, which will be problematic later. Currently, the issue seems to originate from the TensorRT node, and not the decoder.

[mzahana]

@kajananchinniahNV thanks for your response.

My problem was solved after I changed the hard-coded number of classes according to my custom model.

I suggest to make it a parameter.

Thanks.

[kajananchinniahNV]

Thanks for finding a fix to the issue! I'll remove the needs info label and keep the bug label.

We'll be working on a patch to this issue soon. The solution you recommend sounds reasonable to me.

@immaixq could you confirm that this also solves it for you?

[mzahana]

@kajananchinniahNV thank you.

Actually, to make the fix more complete, the output dimension of the model could be a parameter. This way, the decoder can adapt to custom models with output tensor dimensions(not only the number of classes) that is different from the default one by Yolov8 [1, 84, 8400].

Thanks.

[immaixq]

@mzahana Thank you for your help and advice, I have gained some insights after looking into netron again. I did changed the number of classes in the script and I think the error is caused by the number of outputs my model gives, which consists of output0 and output1. I managed to fix the mismatch error after changing output_tensor_names and output_binding_names to [output0, output1]. However, another error occurred which I'm not sure of what could be the cause of it.

@kajananchinniahNV Thank you for your help, I have changed the parameters as mentioned above and managed to fix the mismatch error. Unfortunately another error occurred. The process has died without additional log on its cause. it'd be great if you could share some insights or advice on how to troubleshoot the cause of it.

Really appreciate both of your help, thank you!