Unable to run my own model!

[hh123445]

Description
I have passed the sample program, but I encountered problems when running my own model. The error is as follows:

Triton Information
What version of Triton are you using?
I used Docker to pull the 22.02 version of the triton image. I noticed the issue of the version correspondence between triton and the local tensorRT. According to the official form of nvidia, my local TensorRT uses version 8.2.3.

To Reproduce
The model is the official fast r-cnn model downloaded from mmdetection and converted into an engine file through mmdeploy. The following is its configuration file:

[hh123445]

The gpu used is nvidia's 3080

[nv-kmcgill53]

Hi @hh123445, TensorRT models are specific to the version of TensorRT installed and the model of the GPU which it was created on when they run.

The model is the official fast r-cnn model downloaded from mmdetection and converted into an engine file through mmdeploy

I am unsure whether mmdeploy generates the engine file local to your device or it's simply converting the downloaded model. In the latter case, it could be the downloaded model wasn't generated on a GPU with the same compute capability as your 3080. Further up in the triton logs there should be a more verbose error when loading your Faster_rcnn model. Can you please include that? This will help us root cause.

As a naive solution you can try to build the model yourself on your 3080 with TensorRT 8.2.3 installed and see if that solves your issue.

[hh123445]

Yes, you are right. I converted the model on another computer with different computing power from the 3080, so the model failed to start. When I re-convert the model on the 3080 and deploy it, I can successfully run the model! Thank you!
In addition, the model converted by mmdeploy needs to add mmdeploy special operators to the instructions during deployment. The new commands are as follows:

sudo docker run --gpus all --rm --net=host -p8000:8000 -p8001:8001 -p8002:8002 -v /my_path/triton_server/models:/models -v /my_path/triton_server/plugins:/plugins --env LD_PRELOAD=/plugins/libmmdeploy_tensorrt_ops.so nvcr.io/nvidia/tritonserver:22.02-py3 tritonserver --model-repository=/models

You need to create a new plugins folder outside the models folder and copy the libmmdeploy_tensorrt_ops.so file stored in the MMDeploy/build/lib corresponding to the virtual environment to the plugins folder.