URGENT QUERY: Run Paddle Models with ONNX

[VishyAnand28]

Dear Paddle Community,

I humbly request guidance to run PPOCR V3/v4 models with onnx. I wish to get live inference with images and videos.

Questions:

1. How to run paddleocr model with pip install onnxruntime-gpu?
2. Instead of using this code:python3 tools/infer/predict_system.py, any other way to run and get live results?

Context:: I used the below link: https://github.com/PaddlePaddle/PaddleOCR/blob/main/docs/ppocr/infer_deploy/paddle2onnx.en.md to convert the file into onnx, but I do not want to run through this: python3 tools/infer/predict_system.py since it does not provide live-results.

I would like to run and get instant results like the code with PaddleOCR's API:
from paddleocr import PaddleOCR,draw_ocr
ocr = PaddleOCR(use_angle_cls=True, lang='en') # need to run only once to download and load model into memory
img_path = './imgs_en/img_12.jpg'
result = ocr.ocr(img_path, cls=True)
for idx in range(len(result)):
res = result[idx]
for line in res:
print(line)

I humbly request guidance from @GreatV @WenmuZhou @LDOUBLEV @MissPenguin @tink2123 @UserWangZz and others ........ for guidance.

[GreatV]

To run PaddleOCR models (PPOCR V3/V4) with ONNX and achieve live inference for images and videos, here are the steps and answers to your queries:

---

1. How to run a PaddleOCR model with pip install onnxruntime-gpu?

After converting the PaddleOCR model to ONNX format using Paddle2ONNX, you can load the ONNX model using the ONNX Runtime GPU for inference. Here’s how you can achieve it:

Step 1: Install Required Libraries

Ensure you have the following libraries installed:

pip install paddle2onnx onnxruntime-gpu

Step 2: Convert PaddleOCR Model to ONNX

Follow the steps in the provided document to convert the PPOCR V3/V4 models into ONNX format. For example:

paddle2onnx --model_dir ./inference/en_PP-OCRv3_det_infer \
--model_filename inference.pdmodel \
--params_filename inference.pdiparams \
--save_file ./inference/det_onnx/model.onnx \
--opset_version 11 \
--enable_onnx_checker True

Repeat this for the detection (det), recognition (rec), and classification (cls) models.

Step 3: Perform Inference with ONNXRuntime

Here is an example Python script for loading the ONNX models and performing inference:

import onnxruntime as ort
import cv2
import numpy as np

# Load ONNX models
det_session = ort.InferenceSession('./inference/det_onnx/model.onnx', providers=['CUDAExecutionProvider'])
rec_session = ort.InferenceSession('./inference/rec_onnx/model.onnx', providers=['CUDAExecutionProvider'])
cls_session = ort.InferenceSession('./inference/cls_onnx/model.onnx', providers=['CUDAExecutionProvider'])

def preprocess(image_path):
    # Preprocess the image (reshape, normalize, etc.)
    image = cv2.imread(image_path)
    input_blob = cv2.resize(image, (640, 640))  # Resize based on model requirements
    input_blob = input_blob.transpose(2, 0, 1)  # HWC to CHW
    input_blob = input_blob[np.newaxis, :, :, :].astype('float32')  # Add batch dimension
    return input_blob

# Example for detection inference
image_path = './imgs_en/img_12.jpg'
input_blob = preprocess(image_path)
outputs = det_session.run(None, {'x': input_blob})
print("Detection Results:", outputs)

• Replace the preprocessing steps according to your model's requirements.
• Use the detection output to crop regions of interest (ROIs) for text recognition using the rec_session.

---

2. Alternative to python3 tools/infer/predict_system.py for Live Results

The predict_system.py script is a utility provided by PaddleOCR for batch inference. For live inference (real-time processing of images or videos), you can create a custom script using the ONNX Runtime API. Below is an example for live video processing:

Live Video Text Detection and Recognition

import onnxruntime as ort
import cv2
import numpy as np

# Load ONNX models
det_session = ort.InferenceSession('./inference/det_onnx/model.onnx', providers=['CUDAExecutionProvider'])
rec_session = ort.InferenceSession('./inference/rec_onnx/model.onnx', providers=['CUDAExecutionProvider'])

def preprocess_video_frame(frame):
    # Preprocess frame for detection model
    input_blob = cv2.resize(frame, (640, 640))  # Adjust size based on model
    input_blob = input_blob.transpose(2, 0, 1)  # HWC to CHW
    input_blob = input_blob[np.newaxis, :, :, :].astype('float32')  # Add batch dimension
    return input_blob

def run_inference(frame):
    # Preprocess the frame
    input_blob = preprocess_video_frame(frame)
    
    # Run detection model
    det_output = det_session.run(None, {'x': input_blob})[0]
    # Extract ROIs or bounding boxes from det_output (custom logic needed here)
    
    # Run recognition model on cropped ROIs (example placeholder)
    for roi in detected_rois:  # Replace detected_rois with actual ROI extraction logic
        rec_output = rec_session.run(None, {'x': roi})
        print("Recognition Result:", rec_output)

# Open a video stream (webcam or video file)
cap = cv2.VideoCapture(0)  # Use 0 for webcam, or provide video file path

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Run inference on the current frame
    run_inference(frame)

    # Display the frame
    cv2.imshow('Live OCR', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):  # Press 'q' to quit
        break

cap.release()
cv2.destroyAllWindows()

Key Points:

• Detection Model: Use the detection model to identify text areas in the frame.
• Recognition Model: Crop the detected text areas and pass them to the recognition model for text extraction.
• Real-Time Display: Use OpenCV to display the live results on the video feed.

---

Additional Notes:

1. Dynamic Shape: When converting to ONNX, ensure that dynamic shapes are enabled for better compatibility with varying input dimensions.
2. Optimization: Use tools like OnnxSlim for optimizing ONNX models, reducing latency during inference.
3. Custom Pipelines: If you want more control, design a pipeline where detection, cropping, and recognition are modular, allowing you to tailor it for your specific use case.

By using ONNXRuntime with the GPU, you can achieve fast inference while bypassing the limitations of predict_system.py for real-time applications.

---

Response generated by feifei-bot | chatgpt-4o-latest

[jingsongliujing]

There are two packages, namely onnxruntime (used when there is only a CPU) and onnxruntime - gpu (used for inference on a GPU when a GPU is available). You can choose either pip install onnxruntime or pip install onnxruntime - gpu as you like.

[VishyAnand28]

Thanks for answering, again means a lot.

One can simply use onnxruntime - GPU with the below block of code, right? or should one modify onnxocr/onnx_paddleocr.py for it?

import cv2
import onnxruntime as ort

model = ort.InferenceSession(ONNXPaddleOcr())
img = cv2.imread('./1.jpg')

[jingsongliujing]

git clone https://github.com/jingsongliujing/OnnxOCR.git

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt

python test_ocr.py

You can refer to and use this piece of code：https://github.com/jingsongliujing/OnnxOCR/blob/main/test_ocr.py

[VishyAnand28]

Thanks for answering @jingsongliujing, but the code does not explicitly use onnxruntime-gpu. In this line of code "model = ONNXPaddleOcr(use_angle_cls=True, use_gpu=True)", it would mean that the image or video would be processed by the underlying GPU hardware. I am really confused from the past two days how onnxruntime is explicitly used here. Maybe my understanding is extremely poor but I want to assure you that I completely understand the code and I have looked deep into it. Please explain me how onnxruntime-gpu s coupled during inference.

[VishyAnand28]

@jingsongliujing I have loaded the code you have provided and it works well, thanks for your guidance. One last query, how do I provide path specifically only for text detector and put false for detector, recognizer?

Not sure but the onnx models are somehow loaded and I want to change say det_path to something specific.

Also, I have compatible onnx models with CUDA, CUDNN and when I ran with model = ONNXPaddleOcr(use_angle_cls=True, det_model_dir=det_path, use_gpu=False), I got processing time as 0.6 seconds.
output:
print("providers:", onnxruntime.get_device())
providers: GPU
somehow the provider is GPU even if I put as False.

But when I put model = ONNXPaddleOcr(use_angle_cls=True, det_model_dir=det_path, use_gpu=True),

Processing time: 4.5 seconds
2 0 2 5 - 0 1 - 2 6 1 7 : 1 3 : 2 4 . 8 3 8 7 8 0 9 [ W : o n n x r u n t i m e : , s e s s i o n _ s t a t e . c c : 1 1 3 6 o n n x r u n t i m e : : V e r i f y E a c h N o d e I s A s s i g n e d T o A n E p ] S o m e n o d e s w e r e n o t a s s i g n e d t o t h e p r e f e r r e d e x e c u t i o n p r o v i d e r s w h i c h m a y o r m a y n o t h a v e a n n e g a t i v e i m p a c t o n p e r f o r m a n c e . e . g . O R T e x p l i c i t l y a s s i g n s s h a p e r e l a t e d o p s t o C P U t o i m p r o v e p e r f .
2 0 2 5 - 0 1 - 2 6 1 7 : 1 3 : 2 4 . 8 3 9 3 4 7 9 [ W : o n n x r u n t i m e : , s e s s i o n _ s t a t e . c c : 1 1 3 8 o n n x r u n t i m e : : V e r i f y E a c h N o d e I s A s s i g n e d T o A n E p ] R e r u n n i n g w i t h v e r b o s e o u t p u t o n a n o n - m i n i m a l b u i l d w i l l s h o w n o d e a s s i g n m e n t s .

What is happening over here, can you please tell?