Loss of accuracy in recognizing when exporting a trained model

[Balicki24]

I have achieved ~99% accuracy during training. I want to use my trained model in code, for example, instead of typing the command in the terminal every time.

from paddleocr import PaddleOCR,draw_ocr
ocr = PaddleOCR(rec_model_dir='{your_rec_model_dir}', rec_char_dict_path='{your_rec_char_dict_path}')
img_path = 'img_12.jpg'
result = ocr.ocr(img_path)

But when using this code and the final output model based on my trained model on the training dataset , the accuracy was decreased 9%.

Why does it happen like this?

[Balicki24]

And this is my config file :
Global:
debug: false
use_gpu: true
epoch_num: 50
log_smooth_window: 20
print_batch_step: 10
save_model_dir: ./output/rec_ppocr_v3_distillation
save_epoch_step: 500
eval_batch_step: [0, 500]
cal_metric_during_train: true
pretrained_model: /kaggle/input/rec-checkpoint-37/model/best_accuracy.pdparams
checkpoints:
save_inference_dir:
use_visualdl: false
infer_img: doc/imgs_words/ch/word_1.jpg
character_dict_path: /kaggle/input/ch-dict/chinese_cht_dict.txt
max_text_length: &max_text_length 25
infer_mode: false
use_space_char: true
distributed: true
save_res_path: ./output/rec/predicts_ppocrv3_distillation.txt
d2s_train_image_shape: [3, 48, -1]

Optimizer:
name: Adam
beta1: 0.9
beta2: 0.999
lr:
name: Piecewise
decay_epochs : [30]
values : [0.0004, 0.00005]
warmup_epoch: 5
regularizer:
name: L2
factor: 3.0e-05

Architecture:
model_type: &model_type "rec"
name: DistillationModel
algorithm: Distillation
Models:
Teacher:
pretrained:
freeze_params: false
return_all_feats: true
model_type: *model_type
algorithm: SVTR_LCNet
Transform:
Backbone:
name: MobileNetV1Enhance
scale: 0.5
last_conv_stride: [1, 2]
last_pool_type: avg
last_pool_kernel_size: [2, 2]
Head:
name: MultiHead
head_list:
- CTCHead:
Neck:
name: svtr
dims: 64
depth: 2
hidden_dims: 120
use_guide: True
Head:
fc_decay: 0.00001
- SARHead:
enc_dim: 512
max_text_length: *max_text_length
Student:
pretrained:
freeze_params: false
return_all_feats: true
model_type: *model_type
algorithm: SVTR_LCNet
Transform:
Backbone:
name: MobileNetV1Enhance
scale: 0.5
last_conv_stride: [1, 2]
last_pool_type: avg
last_pool_kernel_size: [2, 2]
Head:
name: MultiHead
head_list:
- CTCHead:
Neck:
name: svtr
dims: 64
depth: 2
hidden_dims: 120
use_guide: True
Head:
fc_decay: 0.00001
- SARHead:
enc_dim: 512
max_text_length: *max_text_length
Loss:
name: CombinedLoss
loss_config_list:

• DistillationDMLLoss:
  weight: 1.0
  act: "softmax"
  use_log: true
  model_name_pairs:
  • ["Student", "Teacher"]
    key: head_out
    multi_head: True
    dis_head: ctc
    name: dml_ctc
• DistillationDMLLoss:
  weight: 0.5
  act: "softmax"
  use_log: true
  model_name_pairs:
  • ["Student", "Teacher"]
    key: head_out
    multi_head: True
    dis_head: sar
    name: dml_sar
• DistillationDistanceLoss:
  weight: 1.0
  mode: "l2"
  model_name_pairs:
  • ["Student", "Teacher"]
    key: backbone_out
• DistillationCTCLoss:
  weight: 1.0
  model_name_list: ["Student", "Teacher"]
  key: head_out
  multi_head: True
• DistillationSARLoss:
  weight: 1.0
  model_name_list: ["Student", "Teacher"]
  key: head_out
  multi_head: True

PostProcess:
name: DistillationCTCLabelDecode
model_name: ["Student", "Teacher"]
key: head_out
multi_head: True

Metric:
name: DistillationMetric
base_metric_name: RecMetric
main_indicator: acc
key: "Student"
ignore_space: False

Train:
dataset:
name: SimpleDataSet
data_dir: /kaggle/input/sumitomo-table/train
ext_op_transform_idx: 1
label_file_list:
- /kaggle/input/sumitomo-table/train/train/label.txt
transforms:
- DecodeImage:
img_mode: BGR
channel_first: false
- RecConAug:
prob: 0.5
ext_data_num: 2
image_shape: [48, 320, 3]
max_text_length: *max_text_length
- RecAug:
- MultiLabelEncode:
- RecResizeImg:
image_shape: [3, 48, 320]
- KeepKeys:
keep_keys:
- image
- label_ctc
- label_sar
- length
- valid_ratio
loader:
shuffle: true
batch_size_per_card: 64
drop_last: true
num_workers: 4
Eval:
dataset:
name: SimpleDataSet
data_dir: /kaggle/input/sumitomo-table/train
label_file_list:
- /kaggle/input/sumitomo-table/train/train/label.txt
transforms:
- DecodeImage:
img_mode: BGR
channel_first: false
- MultiLabelEncode:
- RecResizeImg:
image_shape: [3, 48, 320]
- KeepKeys:
keep_keys:
- image
- label_ctc
- label_sar
- length
- valid_ratio
loader:
shuffle: false
drop_last: false
batch_size_per_card: 64
num_workers: 4

[GreatV]

There are several possible reasons why your trained model loses accuracy after exporting:

1. Inference Configuration Differences
   The way the model is used during training (or testing with infer_rec.py) might be different from how it is used after exporting. Ensure that the preprocessing steps (e.g., image resizing, normalization, data augmentation) remain consistent between training and inference.

2. Export Process Issues
   From the related discussion (link), it appears that others have faced a similar problem where the model's accuracy drops significantly after exporting. Double-check the export_model.py command and ensure that the model and configuration file used for exporting are the same as those used for training.

3. Character Dictionary Mismatch
   When loading the model for inference, you specify rec_char_dict_path. If this dictionary is different from the one used during training, the predictions may not align correctly, resulting in lower accuracy. Ensure that the character dictionary remains the same.

4. Model Weight Issues
   If the pretrained model path specified during export is incorrect or outdated, the exported model might not retain the latest trained weights. Verify that you are using the latest checkpoint (latest or best checkpoint saved during training).

5. Different Inference Settings
   Different batch sizes, beam search settings, or optimization flags during inference can also affect results. Ensure that the inference settings match training settings as closely as possible.

Possible Solutions:

• Recheck the export command:

  python tools/export_model.py -c configs/my-ch_PP-OCRv4_rec_hgnet.yml \
      -o Global.pretrained_model="./output/rec_ppocr_v4_hgnet/latest" \
      Global.save_inference_dir="./output/v4_hgnet_1/"

  Ensure that this command uses the exact configuration file that was used for training.

• Compare accuracy between:

  1. Running infer_rec.py on the trained model.
  2. Running PaddleOCR after exporting.
     If the accuracy differs, try debugging by printing the intermediate results.

• Verify that rec_char_dict_path remains unchanged.

• Try re-exporting the model and double-check whether any warnings appear during the export process.

Would you be able to test these steps and check if the issue persists?

---

Response generated by 🤖 feifei-bot | chatgpt-4o-latest

⚠️ This is an AI-generated response. Please verify any critical information.

[scruz10FAU]

I'm having the same issue. The output is complete nonsense whether I use the last checkpoint or the inference model, but the training said my accuracy was 99% from the beginning. It would not run at all on GPU unless I used the beta version based on this error:
#11530

It seems there is an issue during training because I don't see how it went from something that was not very accurate before to having 99% accuracy on pretrain which was at best 93% using CPU version:

[2025/04/18 16:13:45] ppocr INFO: load pretrain successful from ./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy
[2025/04/18 16:13:45] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 8 iterations
eval model:: 100% 5/5 [00:00<00:00, 14.48it/s]
[2025/04/18 16:13:48] ppocr INFO: cur metric, acc: 0.9933332671111155, norm_edit_dis: 0.9988888889629629, fps: 1387.4304515713588
[2025/04/18 16:13:48] ppocr INFO: save best model is to ./output/v3_en_mobile/best_accuracy
[2025/04/18 16:13:48] ppocr INFO: best metric, acc: 0.9933332671111155, is_float16: False, norm_edit_dis: 0.9988888889629629, fps: 1387.4304515713588, best_epoch: 1

The predicts_ppocrv3_en file gives output with random letters and special characters instead of what should be predicted. Example:

train_data/val/ak1165.png	gN	0.013487335294485092
train_data/val/al1204.png	`mj	0.013073633424937725
train_data/val/al1528.png	0gE	0.013548429124057293
train_data/val/al1662.png	mE	0.014326480217278004
train_data/val/ar1097.png	E	0.013055707328021526

Please resolve this issue because CPU is way too slow to train on this amount of data, and the GPU version is inaccurate or unstable. I am using this line to train using CPU:
python3 tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy Train.loader.batch_size_per_card=16 Eval.loader.batch_size_per_card=16 Global.use_gpu=False Global.epoch_num=100 Global.save_best_model=True Global.eval_batch_step=[0,16] Global.output_dir=./output/en_ppocrv3_rec Global.recog_model_save_dir=./output/en_ppocrv3_rec

This is using GPU:
python3 tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy Train.loader.batch_size_per_card=32 Eval.loader.batch_size_per_card=32 Global.epoch_num=100 Global.save_best_model=True Global.eval_batch_step=[0,8] Global.output_dir=./output/en_ppocrv3_rec Global.recog_model_save_dir=./output/en_ppocrv3_rec

and this was used to generate the inference model:

  -c configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml \
  -o print_config=True \
  -o Global.pretrained_model=output/v3_en_mobile/best_accuracy \
  -o Global.save_inference_dir=./inference/rec```

[GreatV]

you may need try paddlle 3.0

 python -m pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
 # python -m pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/

[scruz10FAU]

This gave the same results as the other paddle 3.0 version from the issue I linked, so it still is not making valid predictions despite the high accuracy during training.

[scruz10FAU]

I found that when I download the inference file and run predictions, they are not as bad, however they are definitely not 90% accurate or above. I am not sure why that is the accuracy declared during training because I am using the same data. The local testing returned 21% accuracy.