Memory usage keeps growing without limit when training PP-OCRv5 rec model with PaddleOCR v3.2.0 on PaddlePaddle 3.1.0/3.2.0

[qyhou]

🔎 Search before asking

• I have searched the PaddleOCR Docs and found no similar bug report.
• I have searched the PaddleOCR Issues and found no similar bug report.
• I have searched the PaddleOCR Discussions and found no similar bug report.

🐛 Bug (问题描述)

When training PP-OCRv5 rec model using PaddleOCR v3.2.0 inside Docker, I noticed that memory usage keeps increasing without limit on the following PaddlePaddle versions:

• PaddlePaddle 3.1.0 (paddlepaddle/paddle:3.1.0-gpu-cuda12.9-cudnn9.9, paddlepaddle/paddle:3.1.0-gpu-cuda12.6-cudnn9.5)
• PaddlePaddle 3.2.0 (paddlepaddle/paddle:3.2.0-gpu-cuda12.9-cudnn9.9, paddlepaddle/paddle:3.2.0-gpu-cuda12.6-cudnn9.5)

The same training setup works normally (no memory leak) on PaddlePaddle 3.0.0 (paddlepaddle/paddle:3.0.0-gpu-cuda12.6-cudnn9.5-trt10.5).

🏃‍♂️ Environment (运行环境)

PaddleOCR v3.2.0 release
PaddlePaddle 3.1.0/3.2.0 docker images
OS: Ubuntu 24.04.3 LTS
CPU: Intel(R) Xeon(R) Platinum 8469C
GPU: H20 96G
Memory: 1.0T

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

python -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c xxx.yml

Training data: ~10M samples (3×48×320)
Memory: eventually occupying 1 TB RAM after 15 epochs

[cuicheng01]

Hi, what kind of hardware you are using? Could you please describe your GPU, CPU, and training configuration in more detail? This information will help us contact the relevant R&D personnel for further investigation.

[qyhou]

Hi, what kind of hardware you are using? Could you please describe your GPU, CPU, and training configuration in more detail? This information will help us contact the relevant R&D personnel for further investigation.

The hardware is described in the "Environment".
The amount of gpu card and training command are provided in the last section.
The size of training data is 8.64M.
Here is the configuration, which is basically consistent with PP-OCRv5_server_rec.yml.

Global:
  model_name: text_rec_202508
  debug: false
  use_gpu: true
  epoch_num: 35
  log_smooth_window: 20
  print_batch_step: 10
  save_model_dir: ../output/text_rec_202508
  save_epoch_step: 1000
  eval_batch_step: [0, 9000]
  cal_metric_during_train: false
  pretrained_model: ../pretrained_models/PP-OCRv5_server_rec_pretrained.pdparams
  checkpoints:
  save_inference_dir:
  use_visualdl: false
  infer_img:
  character_dict_path: ./ppocr/utils/dict/ppocrv5_dict.txt
  max_text_length: &max_text_length 30
  infer_mode: false
  use_space_char: true
  distributed: true
  save_res_path:
  d2s_train_image_shape: [3, 48, 320]

Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    name: Cosine
    learning_rate: 0.0003
    warmup_epoch: 0
  regularizer:
    name: L2
    factor: 3.0e-05

Architecture:
  model_type: rec
  algorithm: SVTR_HGNet
  Transform:
  Backbone:
    name: PPHGNetV2_B4
    text_rec: True
  Head:
    name: MultiHead
    head_list:
      - CTCHead:
          Neck:
            name: svtr
            dims: 120
            depth: 2
            hidden_dims: 120
            kernel_size: [1, 3]
            use_guide: True
          Head:
            fc_decay: 0.00001
      - NRTRHead:
          nrtr_dim: 384
          max_text_length: *max_text_length

Loss:
  name: MultiLoss
  loss_config_list:
    - CTCLoss:
    - NRTRLoss:

PostProcess:
  name: CTCLabelDecode

Metric:
  name: RecMetric
  main_indicator: acc

Train:
  dataset:
    name: MultiScaleDataSet
    ds_width: false
    data_dir: ../train_data/text_rec_202508/
    ext_op_transform_idx: 1
    label_file_list:
    - ../train_data/text_rec_202508/train.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - RecAug:
    - MultiLabelEncode:
        gtc_encode: NRTRLabelEncode
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_gtc
        - length
        - valid_ratio
  sampler:
    name: MultiScaleSampler
    scales: [[320, 32], [320, 48], [320, 64]]
    first_bs: &bs 240
    fix_bs: true
    divided_factor: [8, 16] # w, h
    is_training: True
  loader:
    shuffle: true
    batch_size_per_card: *bs
    drop_last: true
    num_workers: 120
Eval:
  dataset:
    name: SimpleDataSet
    data_dir: ../train_data/text_rec_202508/
    label_file_list:
    - ../train_data/text_rec_202508/val.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - MultiLabelEncode:
        gtc_encode: NRTRLabelEncode
    - RecResizeImg:
        image_shape: [3, 48, 320]
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_gtc
        - length
        - valid_ratio
  loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 240
    num_workers: 60