Does Paddle OCRT support Tensorrt to speed up the model inference?

[huyhoangle86]

Hi

I have built the code successfully and I'm able to convert model to ONNX based on the documentation, but I couldn't find anywhere how do I convert the onnx to Tensorrt (. engine) to use Tensorrt to speed up the inference process? Is there any documentation I can follow? and which config should I enable to use Tensorrt?

Thanks

[GreatV]

Yes, PaddleOCR does support inference acceleration using TensorRT, but the integration workflow is not fully documented end-to-end officially. However, users have successfully converted ONNX models (exported from PaddleOCR) to TensorRT engines to speed up inference.

Here’s how you can proceed:

1. Export PaddleOCR model to ONNX:
   You've already completed this based on your message. Just make sure the exported ONNX model works correctly and is validated.

2. Convert ONNX to TensorRT Engine:
   This can be done using the trtexec tool provided by TensorRT. For example:

trtexec --onnx=rec.onnx --saveEngine=rec.engine --explicitBatch

Note:

• The --explicitBatch flag is important for ONNX models exported with dynamic shapes.
• You might also need to set --fp16 or --int8 based on required precision and hardware support.

3. Run inference using TensorRT engine:
   PaddleOCR by default uses Paddle Inference or OpenVINO backends. To use TensorRT directly with the engine you generated, you will need a custom inference pipeline, since the official PaddleOCR repo does not natively support ingesting .engine files.

For that, you can:

• Integrate TensorRT engine inference using Python bindings (pycuda, tensorrt)
• Or build a C++ inference application using TensorRT SDK

4. Alternative: Use Paddle Inference with TensorRT Backend:
   If you want to stay within the Paddle ecosystem, Paddle Inference supports TensorRT as a backend. You can enable it through config:

From deploy/python/predict_system.py, in the config setup:

config.enable_tensorrt_engine(
workspace_size=1 << 30,
max_batch_size=1,
min_subgraph_size=3,
precision_mode=paddle.inference.PrecisionType.Float32, # or Float16, Int8
use_static=False,
use_calib_mode=False)

Make sure you have built PaddlePaddle with TensorRT support.

References:

• Similar Issue regarding ONNX → TensorRT conversion in PaddleOCR: paddleocr4 onnx模型转engine出现错误咋解决啊 #13818

In summary:

• Yes, TensorRT is supported for inference acceleration.
• You can export ONNX → TensorRT manually using trtexec.
• Direct usage of .engine in PaddleOCR is not officially documented, so using Paddle Inference with TensorRT backend is more straightforward.

Let me know if you need an example of writing an inference script using TensorRT engine.

---

Response generated by 🤖 feifei-bot | chatgpt-4o-latest

⚠️ This is an AI-generated response. Please verify any critical information.