使用官方預訓練模型修改Backbone參數後無法迭代訓練

[cocoyen1995]

各位好，這邊要辨識的資料集主要只有三種內容。使用的預訓練模型是 en_PP-OCRv3_rec
依照預訓練模型初步推論出來的準度，我手動複製原本各種只有4~5張照片的資料集至以下數量:

• 第一種: 500張
• 第二種: 1500張
• 第三種: 3500張 (->因預訓練權重推論效果較差，故增加樣本數)

訓練完之後，第一種及第二種的準度有保持，但第三種的檢測結果部分變差了
具體變差情況大概舉例如下:

• 原始標註內容→ THIS IS EXAMPLE:.00"-.123"
• 原預訓練權重推論結果→ THISISEXOPIE:23"
• 新訓練權重推論結果→ THIS IS EXEXAS .MPLEST" EXSHI:23"

原本是想加強空白跟標點符號的辨識效果，但整個內容跟標點直接亂掉
查找過一些資料有說可能可以調整Backbone
但如果同一個Backbone的模型修改scale(原0.5改0.6)，訓練時會顯示很多這種shape不合的警告，然後從頭訓練(acc為0)

[2024/07/04 09:18:16] ppocr WARNING: The shape of model params backbone.conv1._conv.weight [19, 3, 3, 3] not matched with loaded params backbone.conv1._conv.weight [16, 3, 3, 3] !
[2024/07/04 09:18:16] ppocr WARNING: The shape of model params backbone.conv1._batch_norm.weight [19] not matched with loaded params backbone.conv1._batch_norm.weight [16] !
[2024/07/04 09:18:16] ppocr WARNING: The shape of model params backbone.conv1._batch_norm.bias [19] not matched with loaded params backbone.conv1._batch_norm.bias [16] !
[2024/07/04 09:18:16] ppocr WARNING: The shape of model params backbone.conv1._batch_norm._mean [19] not matched with loaded params backbone.conv1._batch_norm._mean [16] !
[2024/07/04 09:18:16] ppocr WARNING: The shape of model params backbone.conv1._batch_norm._variance [19] not matched with loaded params backbone.conv1._batch_norm._variance [16] !
...
[2024/07/04 09:18:46] ppocr INFO: epoch: [1/200], global_step: 10, lr: 0.000012, acc: 0.000000, norm_edit_dis: 0.001649, CTCLoss: 211.109406, SARLoss: 4.688298, loss: 215.826614, avg_reader_cost: 4.66088 s, avg_batch_cost: 5.62271 s, avg_samples: 64.0, ips: 11.38242 samples/s, eta: 1 day, 0:20:57, max_mem_reserved: 8534 MB, max_mem_allocated: 8216 MB
[2024/07/04 09:19:28] ppocr INFO: epoch: [1/200], global_step: 20, lr: 0.000037, acc: 0.000000, norm_edit_dis: 0.004630, CTCLoss: 186.840942, SARLoss: 4.432905, loss: 191.274864, avg_reader_cost: 4.68278 s, avg_batch_cost: 5.12917 s, avg_samples: 64.0, ips: 12.47766 samples/s, eta: 23:15:57, max_mem_reserved: 8534 MB, max_mem_allocated: 8216 MB

如果直接把Backbone由原本的MobileNetV1Enhance置換成MobileNetV3，也是會顯示一些找不到layer的警告，然後從頭訓練

[2024/07/04 09:10:06] ppocr WARNING: The pretrained params backbone.conv1._conv.weight not in model
[2024/07/04 09:10:06] ppocr WARNING: The pretrained params backbone.conv1._batch_norm.weight not in model
[2024/07/04 09:10:06] ppocr WARNING: The pretrained params backbone.conv1._batch_norm.bias not in model
[2024/07/04 09:10:06] ppocr WARNING: The pretrained params backbone.conv1._batch_norm._mean not in model
[2024/07/04 09:10:06] ppocr WARNING: The pretrained params backbone.conv1._batch_norm._variance not in model
...
[2024/07/04 09:10:39] ppocr INFO: epoch: [1/200], global_step: 10, lr: 0.000012, acc: 0.000000, norm_edit_dis: 0.007060, CTCLoss: 179.825027, SARLoss: 4.824817, loss: 184.578094, avg_reader_cost: 2.74393 s, avg_batch_cost: 3.27979 s, avg_samples: 64.0, ips: 19.51344 samples/s, eta: 14:12:11, max_mem_reserved: 3813 MB, max_mem_allocated: 2949 MB
[2024/07/04 09:11:15] ppocr INFO: epoch: [1/200], global_step: 20, lr: 0.000037, acc: 0.000000, norm_edit_dis: 0.005787, CTCLoss: 159.817200, SARLoss: 4.506645, loss: 164.366013, avg_reader_cost: 3.30379 s, avg_batch_cost: 3.62142 s, avg_samples: 64.0, ips: 17.67261 samples/s, eta: 14:56:00, max_mem_reserved: 3813 MB, max_mem_allocated: 2949 MB

附上上述初始訓練時的設定檔

Global:
  debug: false
  use_gpu: true
  epoch_num: 200
  log_smooth_window: 10
  print_batch_step: 10
  save_model_dir: ./output/rec_ppocr_v3
  save_epoch_step: 50
  eval_batch_step: [0, 100]
  cal_metric_during_train: true
  pretrained_model: ./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy.pdparams
  checkpoints:
  save_inference_dir:
  use_visualdl: false
  infer_img: doc/imgs_words/ch/word_1.jpg
  character_dict_path: ppocr/utils/en_dict.txt
  max_text_length: &max_text_length 35
  infer_mode: false
  use_space_char: true
  distributed: true
  save_res_path: ./output/rec/predicts_ppocrv3_en.txt


Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    name: Cosine
    learning_rate: 0.001
    warmup_epoch: 5
  regularizer:
    name: L2
    factor: 3.0e-05


Architecture:
  model_type: rec
  algorithm: SVTR_LCNet
  Transform:
  Backbone:
    name: MobileNetV1Enhance
    scale: 0.5
    last_conv_stride: [1, 2]
    last_pool_type: avg
    last_pool_kernel_size: [2, 2]
  Head:
    name: MultiHead
    head_list:
      - CTCHead:
          Neck:
            name: svtr
            dims: 64
            depth: 2
            hidden_dims: 120
            use_guide: True
          Head:
            fc_decay: 0.00001
      - SARHead:
          enc_dim: 512
          max_text_length: *max_text_length

Loss:
  name: MultiLoss
  loss_config_list:
    - CTCLoss:
    - SARLoss:

PostProcess:
  name: CTCLabelDecode

Metric:
  name: RecMetric
  main_indicator: acc
  ignore_space: False

Train:
  dataset:
    name: SimpleDataSet
    data_dir: ./train_data/rec/train
    ext_op_transform_idx: 1
    label_file_list:
    - ./train_data/rec/train.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - RecConAug:
        prob: 0.5
        ext_data_num: 2
        image_shape: [48, 320, 3]
        max_text_length: *max_text_length
    - RecAug:
    - MultiLabelEncode:
    - RecResizeImg:
        image_shape: [3, 48, 320]
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_sar
        - length
        - valid_ratio
  loader:
    shuffle: true
    batch_size_per_card: 64
    drop_last: true
    num_workers: 4
Eval:
  dataset:
    name: SimpleDataSet
    data_dir: ./train_data/rec/val
    label_file_list:
    - ./train_data/rec/val.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - MultiLabelEncode:
    - RecResizeImg:
        image_shape: [3, 48, 320]
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_sar
        - length
        - valid_ratio
  loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 1
    num_workers: 4

該次整體訓練log相關數據變化如圖

想請問這種情況下會建議如何調整訓練手法呢？
(以加強標點符號的檢出及對應樣本英數字辨識為目標的話)
先謝謝耐心看到這邊的各位了🙏