RuntimeError: DataLoader 1 workers exit unexpectedly

[batara666]

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

• 系统环境/System Environment：WSL Ubuntu Windows 11
• 版本号/Version：Paddle： PaddleOCR：paddleocr==2.7.0.3 问题相关组件/Related components：paddlepaddle-gpu==2.5.1
• 运行指令/Command Code：python3 PaddleOCR/tools/train.py -c rec_icdar15_train.yml
• 完整报错/Complete Error Message：

W1122 02:51:59.620342 14268 gpu_resources.cc:149] device: 0, cuDNN Version: 8.9.
[2023/11/22 02:52:00] ppocr INFO: train dataloader has 18 iters
[2023/11/22 02:52:00] ppocr INFO: valid dataloader has 18 iters
[2023/11/22 02:52:00] ppocr INFO: train from scratch
[2023/11/22 02:52:00] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations
Exception in thread Thread-1 (_thread_loop):
Traceback (most recent call last):
  File "/home/zen/.local/lib/python3.10/site-packages/paddle/io/dataloader/dataloader_iter.py", line 693, in _get_data
    data = self._data_queue.get(timeout=self._timeout)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 114, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/zen/.local/lib/python3.10/site-packages/paddle/io/dataloader/dataloader_iter.py", line 604, in _thread_loop
    batch = self._get_data()
  File "/home/zen/.local/lib/python3.10/site-packages/paddle/io/dataloader/dataloader_iter.py", line 708, in _get_data
    raise RuntimeError(
RuntimeError: DataLoader 1 workers exit unexpectedly, pids: 14294
Traceback (most recent call last):
  File "/mnt/c/training-ktp/PaddleOCR/tools/train.py", line 227, in <module>
    main(config, device, logger, vdl_writer)
  File "/mnt/c/training-ktp/PaddleOCR/tools/train.py", line 198, in main
    program.train(config, train_dataloader, valid_dataloader, device, model,
  File "/mnt/c/training-ktp/PaddleOCR/tools/program.py", line 269, in train
    for idx, batch in enumerate(train_dataloader):
  File "/home/zen/.local/lib/python3.10/site-packages/paddle/io/dataloader/dataloader_iter.py", line 825, in __next__
    self._reader.read_next_list()[0]
SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception.     
  [Hint: Expected killed_ != true, but received killed_:1 == true:1.] (at ../paddle/fluid/operators/reader/blocking_queue.h:175)

我们提供了AceIssueSolver来帮助你解答问题，你是否想要它来解答(请填写yes/no)?/We provide AceIssueSolver to solve issues, do you want it? (Please write yes/no): what's this

请尽量不要包含图片在问题中/Please try to not include the image in the issue.

rec_icdar15_train.yml:

Global:
  use_gpu: true
  epoch_num: 72
  log_smooth_window: 20
  print_batch_step: 10
  save_model_dir: ./output/rec/ic15/
  save_epoch_step: 3
  # evaluation is run every 2000 iterations
  eval_batch_step: [0, 2000]
  cal_metric_during_train: True
  pretrained_model:
  checkpoints:
  save_inference_dir: ./
  use_visualdl: False
  infer_img: doc/imgs_words_en/word_10.png
  # for data or label process
  character_dict_path: ktp_dictionary.txt
  max_text_length: 25
  infer_mode: False
  use_space_char: False
  save_res_path: ./output/rec/predicts_ic15.txt

Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    learning_rate: 0.001  # Start with a slightly higher learning rate
  regularizer:
    name: 'L2'
    factor: 1e-5  # Introduce a small amount of regularization to prevent overfitting

Architecture:
  model_type: rec
  algorithm: CRNN
  Transform:
  Backbone:
    name: MobileNetV3
    scale: 0.75  # Increase the scale for potentially better feature extraction
    model_name: large
  Neck:
    name: SequenceEncoder
    encoder_type: rnn
    hidden_size: 128  # Increase the hidden size for capturing more complex patterns
  Head:
    name: CTCHead
    fc_decay: 0.00004  # Introduce some decay in the fully connected layer

Loss:
  name: CTCLoss

PostProcess:
  name: CTCLabelDecode

Metric:
  name: RecMetric
  main_indicator: acc

Train:
  dataset:
    name: SimpleDataSet
    data_dir: /mnt/c/training-ktp/
    label_file_list: ["/mnt/c/training-ktp/ktpannotations.txt"]
    transforms:
      - DecodeImage: # load image
          img_mode: BGR
          channel_first: False
      - RecAug: # Data augmentation
      - CTCLabelEncode: # Class handling label
      - RecResizeImg:
          image_shape: [3, 32, 128]  # Dynamic shape can be considered based on your dataset
      - KeepKeys:
          keep_keys: ['image', 'label', 'length'] 
  loader:
    shuffle: True
    batch_size_per_card: 2  # Adjusted for memory constraints and to be a power of 2
    drop_last: True
    num_workers: 1  # Adjust based on your CPU
    use_shared_memory: False  # Enable shared memory for faster data loading

Eval:
  dataset:
    name: SimpleDataSet
    data_dir: /mnt/c/training-ktp/
    label_file_list: ["/mnt/c/training-ktp/ktpannotations.txt"]
    transforms:
      - DecodeImage: # load image
          img_mode: BGR
          channel_first: False
      - CTCLabelEncode: # Class handling label
      - RecResizeImg:
          image_shape: [3, 32, 128]  # Match the training image shape
      - KeepKeys:
          keep_keys: ['image', 'label', 'length'] 
  loader:
    shuffle: False
    drop_last: False
    batch_size_per_card: 2  # Consistent with the training batch size
    num_workers: 1  # Can be lower than training if resources are limited
    use_shared_memory: False  # Enable shared memory for faster data loading

Folder/File structure:

.
├── PaddleOCR
├── auto-nik-cropper.py
├── batch-auto-nik-cropped.py
├── crop_image.png
├── dataset
├── ktp1.jpg
├── ktp_dictionary.txt
├── ktpannotations.txt
├── ktpcropped
├── ktpdukdown.go
├── ktpdukmeta.csv
├── ktppictures
├── label-gen-fn.py
├── output
├── rec_icdar15_train.yml
└── train_labels.txt

ktp annotation file are having value like this:

ktppictures/xxx.jpg	[{"transcription": "xxxxx", "points": [[159.0, 56.0], [412.0, 58.0], [412.0, 78.0], [158.0, 76.0]]}]
ktppictures/xxx.jpg	[{"transcription": "xxxxx", "points": [[159.0, 56.0], [412.0, 58.0], [412.0, 78.0], [158.0, 76.0]]}]
ktppictures/xxx.jpg	[{"transcription": "xxxxx", "points": [[159.0, 56.0], [412.0, 58.0], [412.0, 78.0], [158.0, 76.0]]}]
ktppictures/xxx.jpg	[{"transcription": "xxxxx", "points": [[159.0, 56.0], [412.0, 58.0], [412.0, 78.0], [158.0, 76.0]]}]
ktppictures/xxx.jpg	[{"transcription": "xxxxx", "points": [[159.0, 56.0], [412.0, 58.0], [412.0, 78.0], [158.0, 76.0]]}]
ktppictures/xxx.jpg	[{"transcription": "xxxxx", "points": [[159.0, 56.0], [412.0, 58.0], [412.0, 78.0], [158.0, 76.0]]}]