RE model train error

[c-avan]

paddleOCR2.6版本
cuda：11.2
cudnn:8.1

使用官方数据集可以正常训练，使用自己的数据集训练出现以下问题：
官方的数据集格式的对应关系都为【“question”，“answer”】，我自己的数据集存在三对关系【"A_1","A_2","B_1","B_2","C_1","C_2"】
训练两轮出现这样的问题：

[2023/12/13 02:22:49] ppocr INFO: epoch: [1/200], global_step: 10, lr: 0.000000, loss: 0.620230, avg_reader_cost: 0.01018 s, avg_batch_cost: 0.33232 s, avg_samples: 1.0, ips: 3.00916 samples/s, eta: 3:32:37
[2023/12/13 02:22:51] ppocr INFO: epoch: [1/200], global_step: 20, lr: 0.000000, loss: 0.770040, avg_reader_cost: 0.00027 s, avg_batch_cost: 0.21143 s, avg_samples: 1.0, ips: 4.72965 samples/s, eta: 2:53:54
Traceback (most recent call last):
File "tools/train.py", line 201, in
main(config, device, logger, vdl_writer)
File "tools/train.py", line 176, in main
eval_class, pre_best_model_dict, logger, vdl_writer, scaler,amp_level, amp_custom_black_list)
File "/home/workspace_disk/PaddleOCR-2.6.0/tools/program.py", line 299, in train
preds = model(batch)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/home/workspace_disk/PaddleOCR-2.6.0/ppocr/modeling/architectures/base_model.py", line 86, in forward
x = self.backbone(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/home/workspace_disk/PaddleOCR-2.6.0/ppocr/modeling/backbones/vqa_layoutlm.py", line 233, in forward
relations=x[6])
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 1485, in forward
head_mask=head_mask,
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 992, in forward
output_hidden_states=output_hidden_states,
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 606, in forward
rel_2d_pos=rel_2d_pos,
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 689, in forward
rel_2d_pos=rel_2d_pos,
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 479, in forward
attention_output = self.output(self_outputs[0], hidden_states)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 329, in forward
hidden_states = self.dense(hidden_states)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/nn/layer/common.py", line 172, in forward
x=input, weight=self.weight, bias=self.bias, name=self.name)
File "/usr/local/lib/python3.7/dist-packages/paddle/nn/functional/common.py", line 1542, in linear
False)
OSError: (External) CUBLAS error(13).
[Hint: 'CUBLAS_STATUS_EXECUTION_FAILED'. The GPU program failed to execute. This is often caused by a launch failure of the kernel on the GPU, which can be caused by multiple reasons. To correct: check that the hardware, an appropriate version of the driver, and the cuBLAS library are correctly installed. ] (at /paddle/paddle/phi/kernels/funcs/blas/blas_impl.cu.h:35)
[operator < matmul_v2 > error]
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/dataloader_iter.py", line 218, in _thread_loop
self._thread_done_event)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/fetcher.py", line 134, in fetch
data = self.collate_fn(data)
File "/home/workspace_disk/PaddleOCR-2.6.0/ppocr/data/collate_fn.py", line 57, in call
data_dict[idx] = paddle.to_tensor(data_dict[idx])
File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 433, in impl
), "We only support '%s()' in dynamic graph mode, please call 'paddle.disable_static()' to enter dynamic graph mode." % func.name
AssertionError: We only support 'to_tensor()' in dynamic graph mode, please call 'paddle.disable_static()' to enter dynamic graph mode.