paddleOCR onnx cpu is slower than cpu

[rafikg]

I convert models to onnx using paddleocr_convert package. then I run the following on 200 images:

ONNX-CPU Experiment

PaddleOCR(use_angle_cls=True,
                             det_model_dir='~/.paddleocr/whl/onnx/en_PP-OCRv3_det_infer/en_PP-OCRv3_det_infer.onnx',
                             rec_model_dir='~/.paddleocr/whl/onnx/en_PP-OCRv4_rec_infer/en_PP-OCRv4_rec_infer.onnx',                             
                             cls_model_dir='~/.paddleocr/whl/onnx/ch_ppocr_mobile_v2.0_cls_infer/ch_ppocr_mobile_v2.0_cls_infer.onnx'
                             lang="en,
                             use_gpu=False,
                             use_onnx=True,
                             enable_mkldnn=True,
                             use_tensorrt=False)

Total Inference time 146.5897572040558 seconds

CPU Experiment

PaddleOCR(use_angle_cls=True,
                             det_model_dir='~/.paddleocr/whl/det/en/en_PP-OCRv3_det_infer/',
                             rec_model_dir='~/.paddleocr/whl/rec/en/en_PP-OCRv4_rec_infer',                             
                             cls_model_dir='~/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer/'
                             lang="en,
                             use_gpu=False,
                             use_onnx=False,
                             enable_mkldnn=True,
                             use_tensorrt=False)

Total Inference time 69.86295700073242 seconds

GPU Experiment

PaddleOCR(use_angle_cls=True,
                             det_model_dir='~/.paddleocr/whl/det/en/en_PP-OCRv3_det_infer/',
                             rec_model_dir='~/.paddleocr/whl/rec/en/en_PP-OCRv4_rec_infer',                             
                             cls_model_dir='~/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer/'
                             lang="en,
                             use_gpu=True,
                             use_onnx=False,
                             enable_mkldnn=True,
                             use_tensorrt=False)

Total Inference time 25.379971742630005

TensorRT Experiment

PaddleOCR(use_angle_cls=True,
                             det_model_dir='~/.paddleocr/whl/det/en/en_PP-OCRv3_det_infer/',
                             rec_model_dir='~/.paddleocr/whl/rec/en/en_PP-OCRv4_rec_infer',                             
                             cls_model_dir='~/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer/'
                             lang="en,
                             use_gpu=True,
                             use_onnx=False,
                             enable_mkldnn=False,
                             use_tensorrt=True)

Inference time 162.79134917259216

I expect that:

1. ONNX-CPU inference time should be less than CPU Time.
2. TensorRT time should be less than GPU time

Did I miss anything ?

[GreatV]

Are the image shapes the same?

[rafikg]

@GreatV nope, the shape is not static. I know that onnx with GPU is slow if the shape is dynamic. I read about TensorRT and it has the same problem. One solution is to reshape all the images to the same shape.

Is there any other solution? For example, other parameters to specify in order to deal effectively with dynamic shape?