[Error] getting dataset loader when trainning pp-OCRv4

[phamkhactu]

• System Environment：
• Version：Paddle： PaddleOCR 2.7
• Command Code：python3 tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml
• Complete Error Message：

un every 2000 iterations
Exception in thread Thread-7:
Traceback (most recent call last):
  File "/home/tupk/anaconda3/envs/ocr_v2/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/tupk/anaconda3/envs/ocr_v2/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tupk/anaconda3/envs/ocr_v2/lib/python3.7/site-packages/paddle/io/dataloader/dataloader_iter.py", line 604, in _thread_loop
    batch = self._get_data()
  File "/home/tupk/anaconda3/envs/ocr_v2/lib/python3.7/site-packages/paddle/io/dataloader/dataloader_iter.py", line 752, in _get_data
    batch.reraise()
  File "/home/tupk/anaconda3/envs/ocr_v2/lib/python3.7/site-packages/paddle/io/dataloader/worker.py", line 178, in reraise
    raise self.exc_type(msg)
IndexError: DataLoader worker(0) caught IndexError with message:
Traceback (most recent call last):
  File "/home/tupk/anaconda3/envs/ocr_v2/lib/python3.7/site-packages/paddle/io/dataloader/worker.py", line 363, in _worker_loop
    batch = fetcher.fetch(indices)
  File "/home/tupk/anaconda3/envs/ocr_v2/lib/python3.7/site-packages/paddle/io/dataloader/fetcher.py", line 78, in fetch
    data.append(self.dataset[idx])
  File "/home/tupk/tupk/ocr-digits/PaddleOCR/ppocr/data/lmdb_dataset.py", line 135, in __getitem__
    lmdb_idx, file_idx = self.data_idx_order_list[idx]
IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

When I debug, I saw that in lmdb_dataset.py with function:

    def __getitem__(self, idx):
        lmdb_idx, file_idx = self.data_idx_order_list[idx]
        lmdb_idx = int(lmdb_idx)
        file_idx = int(file_idx)
        sample_info = self.get_lmdb_sample_info(self.lmdb_sets[lmdb_idx]['txn'],
                                                file_idx)
        if sample_info is None:
            return self.__getitem__(np.random.randint(self.__len__()))
        img, label = sample_info
        data = {'image': img, 'label': label}
        data['ext_data'] = self.get_ext_data()
        outs = transform(data, self.ops)
        if outs is None:
            return self.__getitem__(np.random.randint(self.__len__()))
        return outs

idx always returns with 4 values: (number1, number2, number3, None), but I think idx value should be a single value.

[tink2123]

The training set supports two data types: simple_data and lmdb_data
If you use en_PP-OCRv4_rec.yml, simple_data is used by default. Please note that in the label file, please separate the image path and image label with \t by default. If you use other methods to separate it, it will cause a training error.

"Image file name image annotation information"

train_data/rec/train/word_001.jpg ABCDE
train_data/rec/train/word_002.jpg 12345
...

[phamkhactu]

@tink2123 I used lmdb_data. Sometimes, idx in getitem returned 3 value [32,320,1868], [48,320,4651],[64,320,4651], [5645]

I tried to get idx = idx[2] if len(idx) > 1 else: idx=idx[0].

I think that it causes from sampler = MultiScaleSampler, scales: [[320, 32], [320, 48], [320, 64]]
Would you mind to check it again?