Support for Converting PaddlePaddleOCR (CPU-based) to OpenVINO for Acceleration

[mkrzywda]

I’m currently using PaddlePaddleOCR with a CPU-based setup and would like to accelerate its performance using OpenVINO. My current implementation relies on the following configuration:

ocr = PaddleOCR(
    use_mkldnn=True,
    cpu_threads=16,
    use_angle_cls=False,
    lang="en",
    ocr_version="PP-OCR",
    use_gpu=False,
    use_space_char=True,
    det_db_thresh=0.3,
    det_db_box_thresh=0.5,
    cls=False,
    bin=False,
    inv=False,
    alpha_color=False,
    max_text_length=100
)

I’ve explored the possibility of converting the model from PaddlePaddle to ONNX and then from ONNX to OpenVINO, but this approach seems overly complex and potentially inefficient. I’d like to know if there’s a more straightforward or optimized way to convert and run my PaddlePaddleOCR model with OpenVINO acceleration on CPU.

What I Need:

• Guidance or a tool to directly convert my PaddlePaddleOCR model to OpenVINO format.
• Steps to integrate and run the converted model with OpenVINO for CPU acceleration.
• Any best practices or recommendations to optimize performance in this setup.

My system:

pyclipper==1.3.0.post5
attrdict==2.0.1
albucore==0.0.16
albumentations==1.4.14
paddlepaddle==3.0.0b2
paddleocr==2.9.0
grpcio==1.68.0
grpcio-tools==1.68.0
prometheus_client==0.21.0
ruff==0.9.5
openvino
numpy
opencv-python
onnxruntime
paddle2onnx
onnxslim

[GreatV]

To accelerate your PaddlePaddleOCR model using OpenVINO while keeping a CPU-based setup, follow these optimized steps:

1. Convert PaddlePaddle Model to OpenVINO Format

Since OpenVINO does not support PaddlePaddle models natively, the recommended conversion path is:
PaddlePaddle → ONNX → OpenVINO

Step 1: Export the Inference Model from PaddleOCR

Ensure your PaddlePaddle model is exported to an inference format using:

python tools/export_model.py \
  -c path/to/your/config.yml \
  -o Global.pretrained_model=path/to/best_model \
  Global.save_inference_dir=path/to/inference_model

This will create:

• inference.pdmodel
• inference.pdiparams

Step 2: Convert Paddle Model to ONNX

Use paddle2onnx for conversion:

paddle2onnx \
 --model_dir path/to/inference_model \
 --model_filename inference.pdmodel \
 --params_filename inference.pdiparams \
 --opset_version 11 \
 --save_file ocr_model.onnx

Step 3: Convert ONNX Model to OpenVINO IR

Use OpenVINO's Model Optimizer:

mo --input_model ocr_model.onnx --output_dir openvino_model

This generates OpenVINO Intermediate Representation (IR) files:

• ocr_model.xml
• ocr_model.bin

---

2. Run the Model with OpenVINO for Inference

Use OpenVINO's inference engine to load and run your model efficiently.

from openvino.runtime import Core
import numpy as np
import cv2

ie = Core()
model = ie.read_model(model="openvino_model/ocr_model.xml")
compiled_model = ie.compile_model(model=model, device_name="CPU")

# Load an example image
image = cv2.imread("example.jpg")
image = preprocess(image)  # Ensure preprocessing matches the original PaddleOCR requirements

# Perform inference
output = compiled_model([image])
print(output)

---

3. Optimization Tips for Better Performance

1. Use OpenVINO-Specific Optimizations

   • Try different inference precision modes (FP32, FP16):

     mo --input_model ocr_model.onnx --data_type FP16 --output_dir openvino_model

   • Enable CPU_THROUGHPUT_STREAMS=CPU_AUTO and INFERENCE_NUM_THREADS=16 for optimal CPU utilization.

2. Run Benchmarking to Validate Speed Improvements

   benchmark_app -m openvino_model/ocr_model.xml -d CPU -niter 100

3. Optimize ONNX with ONNX Runtime Before Conversion

   • Use ONNX Graph Optimization to remove bottlenecks before converting to OpenVINO:

     import onnx
     from onnxruntime.tools import optimizer
     
     onnx_model = onnx.load("ocr_model.onnx")
     optimized_model = optimizer.optimize(onnx_model, ["eliminate_deadend", "eliminate_identity"])
     onnx.save(optimized_model, "ocr_model_optimized.onnx")

---

Conclusion

Following this structured workflow, you can efficiently run PaddlePaddleOCR on CPU with OpenVINO acceleration. The primary speedup will come from using OpenVINO's inference engine optimized for Intel CPUs.

Would you like specific benchmarks comparing PaddlePaddle MKL-DNN and OpenVINO performance after conversion? 🚀

---

Response generated by 🤖 feifei-bot | chatgpt-4o-latest

⚠️ This is an AI-generated response. Please verify any critical information.

[mkrzywda]

Alternatively, if it can be seamlessly converted to OpenVino from PaddlePaddle, that would be an added advantage.