Training Stops After One Epoch - Need Help

[MatthiasPSW]

I'm having an issue with training (fine-tuning) my PaddleOCR model. The training stops after just one epoch, even though I have configured it to run for 350 epochs. Furthermore the "trained" model is not saved. Here are some details about my configuration and the steps I've already taken:

Global:
  use_amp: true
  amp_level: O2
  amp_dtype: float16
  debug: true
  use_gpu: true
  epoch_num: 350
  log_smooth_window: 20
  print_batch_step: 10
  save_model_dir: C:\Users\XXXXXXX\PycharmProjects\textRecognition\runs\ch_PP-OCR_V3_det\
  save_epoch_step: 100
  eval_batch_step:
    - 0
    - 200
  cal_metric_during_train: false
  pretrained_model: C:\Users\XXXXXXX\PycharmProjects\textRecognition\models\pre_trained\detv3\student.pdparams
  checkpoints: null
  save_inference_dir: null
  use_visualdl: false
  infer_img: doc/imgs_en/img_10.jpg
  save_res_path: C:\Users\XXXXXXX\PycharmProjects\textRecognition\runs\ch_PP-OCR_V3_det\checkpoints
  distributed: false

profiler_options:
  enabled: False

Architecture:
  model_type: det
  algorithm: DB
  Transform:
  Backbone:
    name: MobileNetV3
    scale: 0.5
    model_name: large
    disable_se: True
  Neck:
    name: RSEFPN
    out_channels: 96
    shortcut: True
  Head:
    name: DBHead
    k: 50

Loss:
  name: DBLoss
  balance_loss: true
  main_loss_type: DiceLoss
  alpha: 5
  beta: 10
  ohem_ratio: 3
Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    name: Cosine
    learning_rate: 0.001
    warmup_epoch: 2
  regularizer:
    name: L2
    factor: 5.0e-05
PostProcess:
  name: DBPostProcess
  thresh: 0.3
  box_thresh: 0.6
  max_candidates: 1000
  unclip_ratio: 1.5
Metric:
  name: DetMetric
  main_indicator: hmean
Train:
  dataset:
    name: SimpleDataSet
    data_dir: C:\Users\XXXXXXX\Desktop\label_test\eigener_test\train
    label_file_list:
      - C:\Users\XXXXXXX\Desktop\label_test\eigener_test\train\train_labels.txt
    ratio_list: [1.0]
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - DetLabelEncode: null
    - IaaAugment:
        augmenter_args:
        - type: Fliplr
          args:
            p: 0.5
        - type: Affine
          args:
            rotate:
            - -10
            - 10
        - type: Resize
          args:
            size:
            - 0.5
            - 3
    - EastRandomCropData:
        size:
        - 320
        - 320
        max_tries: 50
        keep_ratio: true
    - MakeBorderMap:
        shrink_ratio: 0.4
        thresh_min: 0.3
        thresh_max: 0.7
    - MakeShrinkMap:
        shrink_ratio: 0.4
        min_text_size: 8
    - NormalizeImage:
        scale: 1./255.
        mean:
        - 0.485
        - 0.456
        - 0.406
        std:
        - 0.229
        - 0.224
        - 0.225
        order: hwc
    - ToCHWImage: null
    - KeepKeys:
        keep_keys:
        - image
        - threshold_map
        - threshold_mask
        - shrink_map
        - shrink_mask
  loader:
    shuffle: true
    drop_last: false
    batch_size_per_card: 1
    num_workers: 2
Eval:
  dataset:
    name: SimpleDataSet
    data_dir: C:\Users\XXXXXXX\Desktop\label_test\eigener_test\val
    label_file_list:
      - C:\Users\XXXXXXX\Desktop\label_test\eigener_test\val\val_labels.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - DetLabelEncode: null
    - DetResizeForTest: null
    - NormalizeImage:
        scale: 1./255.
        mean:
        - 0.485
        - 0.456
        - 0.406
        std:
        - 0.229
        - 0.224
        - 0.225
        order: hwc
    - ToCHWImage: null
    - KeepKeys:
        keep_keys:
        - image
        - shape
        - polys
        - ignore_tags
  loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 1
    num_workers: 1

Log Output

[2025/02/10 14:22:45] ppocr INFO: Initialize indexs of datasets:['C:\\Users\\D5NDU7H\\Desktop\\label_test\\eigener_test\\train\\train_labels.txt']
[2025/02/10 14:22:45] ppocr INFO: Initialize indexs of datasets:['C:\\Users\\D5NDU7H\\Desktop\\label_test\\eigener_test\\val\\val_labels.txt']
[2025/02/10 14:22:45] ppocr INFO: train dataloader has 2000 iters
[2025/02/10 14:22:45] ppocr INFO: valid dataloader has 500 iters
[2025/02/10 14:22:45] ppocr INFO: load pretrain successful from C:\Users\D5NDU7H\PycharmProjects\textRecognition\models\pre_trained\detv3\student
[2025/02/10 14:22:45] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 200 iterations
[2025/02/10 14:22:47] ppocr INFO: epoch: [1/350], global_step: 10, lr: 0.000001, loss: 4.875520, loss_shrink_maps: 3.281333, loss_threshold_maps: 0.862104, loss_binary_maps: 0.656189, loss_cbn: 0.000000, avg_reader_cost: 0.00917 s, avg_batch_cost: 0.11200 s, avg_samples: 2.0, ips: 17.85748 samples/s, eta: 21:46:37, max_mem_reserved: 675 MB, max_mem_allocated: 598 MB
[2025/02/10 14:22:47] ppocr INFO: epoch: [1/350], global_step: 20, lr: 0.000002, loss: 6.075370, loss_shrink_maps: 4.129720, loss_threshold_maps: 0.876745, loss_binary_maps: 0.826083, loss_cbn: 0.000000, avg_reader_cost: 0.00060 s, avg_batch_cost: 0.05556 s, avg_samples: 2.0, ips: 35.99779 samples/s, eta: 16:17:23, max_mem_reserved: 675 MB, max_mem_allocated: 598 MB
============================================Perf Summary============================================
Reader Ratio: 0.059%
Time Unit: s, IPS Unit: steps/s
|                 |       avg       |       max       |       min       |
|   reader_cost   |     0.00006     |     0.00006     |     0.00005     |
|    batch_cost   |     0.09291     |     0.08810     |     0.08250     |
|       ips       |     10.76266    |     12.12121    |     11.35061    |

Process finished with exit code 0

Steps I've Already Taken:

1. Reduced batch size and image size to lower GPU usage.
2. Enabled Mixed Precision Training (use_amp: true).
3. Monitored GPU usage with nvidia-smi and ensured no other processes are using the GPU.

Despite these steps, the training still stops after one epoch. Any help or suggestions on how to resolve this issue would be greatly appreciated!

Thank you!

[GreatV]

It looks like your PaddleOCR training stops after just one epoch despite being configured for 350 epochs. Here are a few possible causes and solutions to troubleshoot the issue:

Possible Causes and Solutions:

1. Training Process Exits Normally

   • Your log output shows Process finished with exit code 0, which indicates that the training process completed successfully without errors.
   • This suggests that the training loop may have been unintentionally configured to stop after one epoch.

2. Check Dataset Size and Iterations

   • Your log indicates that the training data loader has 2000 iterations and the validation data loader has 500 iterations.
   • Ensure that your dataset has enough samples and that the training loop is correctly iterating over all epochs.

3. Verify epoch_num Parameter

   • Your configuration includes epoch_num: 350, which should ensure 350 epochs.
   • Double-check that this value is correctly passed to the training script. Some training scripts may override the configuration file.

4. Check for Early Stopping or Exit Conditions

   • Some training scripts have built-in stopping conditions such as:
     • Reaching a certain validation accuracy
     • Encountering NaN values in the loss
   • Check if any early stopping logic is implemented in your training script.

5. Ensure checkpoints Parameter is Not Triggering a Resume

   • The checkpoints parameter is set to null, meaning training should start from scratch.
   • If you previously trained and stopped at 1 epoch, try deleting any existing checkpoint files in the save_res_path directory before restarting.

6. Debug Using debug: true

   • Since debug: true is enabled, check the logs for any messages that might indicate why training stopped prematurely.

7. Try Running with use_amp: false

   • Mixed-precision training (use_amp: true) can sometimes cause unexpected behavior, especially if the GPU does not support certain precision levels.
   • Try setting use_amp: false and see if it makes a difference.

8. Check GPU Memory and Logs

   • Your GPU memory usage appears low (max_mem_reserved: 675 MB), so insufficient memory is unlikely to be the issue.
   • However, check system logs (dmesg on Linux or Event Viewer on Windows) for any GPU-related errors.

Next Steps:

• Run the training with debug: true and inspect the logs.
• Try setting use_amp: false to see if mixed-precision training is causing issues.
• Ensure no early stopping conditions are active.
• Delete existing checkpoints and start fresh.

If the issue persists, try running with a different dataset or a minimal configuration to isolate the problem.

Let me know if you need further clarification! 🚀

---

Response generated by feifei-bot | chatgpt-4o-latest