【发票关系抽取训练】 按照文档调整配置文件，训练数据使用大礼包内的数据。但是训练的时候发生异常，请Paddle哥看看，完整信息如下：

[gomatthew]

[2024/06/06 03:38:02] ppocr INFO: Architecture :
[2024/06/06 03:38:02] ppocr INFO: Models :
[2024/06/06 03:38:02] ppocr INFO: Student :
[2024/06/06 03:38:02] ppocr INFO: Backbone :
[2024/06/06 03:38:02] ppocr INFO: checkpoints : None
[2024/06/06 03:38:02] ppocr INFO: mode : vi
[2024/06/06 03:38:02] ppocr INFO: name : LayoutXLMForRe
[2024/06/06 03:38:02] ppocr INFO: pretrained : True
[2024/06/06 03:38:02] ppocr INFO: Transform : None
[2024/06/06 03:38:02] ppocr INFO: algorithm : LayoutXLM
[2024/06/06 03:38:02] ppocr INFO: freeze_params : False
[2024/06/06 03:38:02] ppocr INFO: model_type : kie
[2024/06/06 03:38:02] ppocr INFO: pretrained : None
[2024/06/06 03:38:02] ppocr INFO: return_all_feats : True
[2024/06/06 03:38:02] ppocr INFO: Teacher :
[2024/06/06 03:38:02] ppocr INFO: Backbone :
[2024/06/06 03:38:02] ppocr INFO: checkpoints : None
[2024/06/06 03:38:02] ppocr INFO: mode : vi
[2024/06/06 03:38:02] ppocr INFO: name : LayoutXLMForRe
[2024/06/06 03:38:02] ppocr INFO: pretrained : True
[2024/06/06 03:38:02] ppocr INFO: Transform : None
[2024/06/06 03:38:02] ppocr INFO: algorithm : LayoutXLM
[2024/06/06 03:38:02] ppocr INFO: freeze_params : False
[2024/06/06 03:38:02] ppocr INFO: model_type : kie
[2024/06/06 03:38:02] ppocr INFO: pretrained : None
[2024/06/06 03:38:02] ppocr INFO: return_all_feats : True
[2024/06/06 03:38:02] ppocr INFO: algorithm : Distillation
[2024/06/06 03:38:02] ppocr INFO: model_type : kie
[2024/06/06 03:38:02] ppocr INFO: name : DistillationModel
[2024/06/06 03:38:02] ppocr INFO: Eval :
[2024/06/06 03:38:02] ppocr INFO: dataset :
[2024/06/06 03:38:02] ppocr INFO: data_dir : train_data/zzsfp/imgs
[2024/06/06 03:38:02] ppocr INFO: label_file_list : ['train_data/zzsfp/val.json']
[2024/06/06 03:38:02] ppocr INFO: name : SimpleDataSet
[2024/06/06 03:38:02] ppocr INFO: transforms :
[2024/06/06 03:38:02] ppocr INFO: DecodeImage :
[2024/06/06 03:38:02] ppocr INFO: channel_first : False
[2024/06/06 03:38:02] ppocr INFO: img_mode : RGB
[2024/06/06 03:38:02] ppocr INFO: VQATokenLabelEncode :
[2024/06/06 03:38:02] ppocr INFO: algorithm : LayoutXLM
[2024/06/06 03:38:02] ppocr INFO: class_path : train_data/zzsfp/class_list.txt
[2024/06/06 03:38:02] ppocr INFO: contains_re : True
[2024/06/06 03:38:02] ppocr INFO: order_method : tb-yx
[2024/06/06 03:38:02] ppocr INFO: use_textline_bbox_info : True
[2024/06/06 03:38:02] ppocr INFO: VQATokenPad :
[2024/06/06 03:38:02] ppocr INFO: max_seq_len : 512
[2024/06/06 03:38:02] ppocr INFO: return_attention_mask : True
[2024/06/06 03:38:02] ppocr INFO: VQAReTokenRelation : None
[2024/06/06 03:38:02] ppocr INFO: VQAReTokenChunk :
[2024/06/06 03:38:02] ppocr INFO: max_seq_len : 512
[2024/06/06 03:38:02] ppocr INFO: TensorizeEntitiesRelations : None
[2024/06/06 03:38:02] ppocr INFO: Resize :
[2024/06/06 03:38:02] ppocr INFO: size : [224, 224]
[2024/06/06 03:38:02] ppocr INFO: NormalizeImage :
[2024/06/06 03:38:02] ppocr INFO: mean : [123.675, 116.28, 103.53]
[2024/06/06 03:38:02] ppocr INFO: order : hwc
[2024/06/06 03:38:02] ppocr INFO: scale : 1
[2024/06/06 03:38:02] ppocr INFO: std : [58.395, 57.12, 57.375]
[2024/06/06 03:38:02] ppocr INFO: ToCHWImage : None
[2024/06/06 03:38:02] ppocr INFO: KeepKeys :
[2024/06/06 03:38:02] ppocr INFO: keep_keys : ['input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'entities', 'relations']
[2024/06/06 03:38:02] ppocr INFO: loader :
[2024/06/06 03:38:02] ppocr INFO: batch_size_per_card : 8
[2024/06/06 03:38:02] ppocr INFO: drop_last : False
[2024/06/06 03:38:02] ppocr INFO: num_workers : 8
[2024/06/06 03:38:02] ppocr INFO: shuffle : False
[2024/06/06 03:38:02] ppocr INFO: Global :
[2024/06/06 03:38:02] ppocr INFO: cal_metric_during_train : False
[2024/06/06 03:38:02] ppocr INFO: distributed : False
[2024/06/06 03:38:02] ppocr INFO: epoch_num : 130
[2024/06/06 03:38:02] ppocr INFO: eval_batch_step : [0, 19]
[2024/06/06 03:38:02] ppocr INFO: infer_img : ppstructure/docs/kie/input/zh_val_21.jpg
[2024/06/06 03:38:02] ppocr INFO: log_smooth_window : 10
[2024/06/06 03:38:02] ppocr INFO: print_batch_step : 10
[2024/06/06 03:38:02] ppocr INFO: save_epoch_step : 2000
[2024/06/06 03:38:02] ppocr INFO: save_inference_dir : None
[2024/06/06 03:38:02] ppocr INFO: save_model_dir : ./output/re_vi_layoutxlm_xfund_zh_udml
[2024/06/06 03:38:02] ppocr INFO: save_res_path : ./output/re/xfund_zh/with_gt
[2024/06/06 03:38:02] ppocr INFO: seed : 2022
[2024/06/06 03:38:02] ppocr INFO: use_gpu : True
[2024/06/06 03:38:02] ppocr INFO: use_visualdl : False
[2024/06/06 03:38:02] ppocr INFO: Loss :
[2024/06/06 03:38:02] ppocr INFO: loss_config_list :
[2024/06/06 03:38:02] ppocr INFO: DistillationLossFromOutput :
[2024/06/06 03:38:02] ppocr INFO: key : loss
[2024/06/06 03:38:02] ppocr INFO: model_name_list : ['Student', 'Teacher']
[2024/06/06 03:38:02] ppocr INFO: reduction : mean
[2024/06/06 03:38:02] ppocr INFO: weight : 1.0
[2024/06/06 03:38:02] ppocr INFO: DistillationVQADistanceLoss :
[2024/06/06 03:38:02] ppocr INFO: index : 5
[2024/06/06 03:38:02] ppocr INFO: key : hidden_states
[2024/06/06 03:38:02] ppocr INFO: mode : l2
[2024/06/06 03:38:02] ppocr INFO: model_name_pairs : [['Student', 'Teacher']]
[2024/06/06 03:38:02] ppocr INFO: name : loss_5
[2024/06/06 03:38:02] ppocr INFO: weight : 0.5
[2024/06/06 03:38:02] ppocr INFO: DistillationVQADistanceLoss :
[2024/06/06 03:38:02] ppocr INFO: index : 8
[2024/06/06 03:38:02] ppocr INFO: key : hidden_states
[2024/06/06 03:38:02] ppocr INFO: mode : l2
[2024/06/06 03:38:02] ppocr INFO: model_name_pairs : [['Student', 'Teacher']]
[2024/06/06 03:38:02] ppocr INFO: name : loss_8
[2024/06/06 03:38:02] ppocr INFO: weight : 0.5
[2024/06/06 03:38:02] ppocr INFO: name : CombinedLoss
[2024/06/06 03:38:02] ppocr INFO: Metric :
[2024/06/06 03:38:02] ppocr INFO: base_metric_name : VQAReTokenMetric
[2024/06/06 03:38:02] ppocr INFO: key : Student
[2024/06/06 03:38:02] ppocr INFO: main_indicator : hmean
[2024/06/06 03:38:02] ppocr INFO: name : DistillationMetric
[2024/06/06 03:38:02] ppocr INFO: Optimizer :
[2024/06/06 03:38:02] ppocr INFO: beta1 : 0.9
[2024/06/06 03:38:02] ppocr INFO: beta2 : 0.999
[2024/06/06 03:38:02] ppocr INFO: clip_norm : 10
[2024/06/06 03:38:02] ppocr INFO: lr :
[2024/06/06 03:38:02] ppocr INFO: learning_rate : 5e-05
[2024/06/06 03:38:02] ppocr INFO: warmup_epoch : 10
[2024/06/06 03:38:02] ppocr INFO: name : AdamW
[2024/06/06 03:38:02] ppocr INFO: regularizer :
[2024/06/06 03:38:02] ppocr INFO: factor : 0.0
[2024/06/06 03:38:02] ppocr INFO: name : L2
[2024/06/06 03:38:02] ppocr INFO: PostProcess :
[2024/06/06 03:38:02] ppocr INFO: key : None
[2024/06/06 03:38:02] ppocr INFO: model_name : ['Student', 'Teacher']
[2024/06/06 03:38:02] ppocr INFO: name : DistillationRePostProcess
[2024/06/06 03:38:02] ppocr INFO: Train :
[2024/06/06 03:38:02] ppocr INFO: dataset :
[2024/06/06 03:38:02] ppocr INFO: data_dir : train_data/zzsfp/imgs
[2024/06/06 03:38:02] ppocr INFO: label_file_list : ['train_data/zzsfp/train.json']
[2024/06/06 03:38:02] ppocr INFO: name : SimpleDataSet
[2024/06/06 03:38:02] ppocr INFO: ratio_list : [1.0]
[2024/06/06 03:38:02] ppocr INFO: transforms :
[2024/06/06 03:38:02] ppocr INFO: DecodeImage :
[2024/06/06 03:38:02] ppocr INFO: channel_first : False
[2024/06/06 03:38:02] ppocr INFO: img_mode : RGB
[2024/06/06 03:38:02] ppocr INFO: VQATokenLabelEncode :
[2024/06/06 03:38:02] ppocr INFO: algorithm : LayoutXLM
[2024/06/06 03:38:02] ppocr INFO: class_path : train_data/zzsfp/class_list.txt
[2024/06/06 03:38:02] ppocr INFO: contains_re : True
[2024/06/06 03:38:02] ppocr INFO: order_method : tb-yx
[2024/06/06 03:38:02] ppocr INFO: use_textline_bbox_info : True
[2024/06/06 03:38:02] ppocr INFO: VQATokenPad :
[2024/06/06 03:38:02] ppocr INFO: max_seq_len : 512
[2024/06/06 03:38:02] ppocr INFO: return_attention_mask : True
[2024/06/06 03:38:02] ppocr INFO: VQAReTokenRelation : None
[2024/06/06 03:38:02] ppocr INFO: VQAReTokenChunk :
[2024/06/06 03:38:02] ppocr INFO: max_seq_len : 512
[2024/06/06 03:38:02] ppocr INFO: TensorizeEntitiesRelations : None
[2024/06/06 03:38:02] ppocr INFO: Resize :
[2024/06/06 03:38:02] ppocr INFO: size : [224, 224]
[2024/06/06 03:38:02] ppocr INFO: NormalizeImage :
[2024/06/06 03:38:02] ppocr INFO: mean : [123.675, 116.28, 103.53]
[2024/06/06 03:38:02] ppocr INFO: order : hwc
[2024/06/06 03:38:02] ppocr INFO: scale : 1
[2024/06/06 03:38:02] ppocr INFO: std : [58.395, 57.12, 57.375]
[2024/06/06 03:38:02] ppocr INFO: ToCHWImage : None
[2024/06/06 03:38:02] ppocr INFO: KeepKeys :
[2024/06/06 03:38:02] ppocr INFO: keep_keys : ['input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'entities', 'relations']
[2024/06/06 03:38:02] ppocr INFO: loader :
[2024/06/06 03:38:02] ppocr INFO: batch_size_per_card : 2
[2024/06/06 03:38:02] ppocr INFO: drop_last : False
[2024/06/06 03:38:02] ppocr INFO: num_workers : 4
[2024/06/06 03:38:02] ppocr INFO: shuffle : True
[2024/06/06 03:38:02] ppocr INFO: profiler_options : None
[2024/06/06 03:38:02] ppocr INFO: train with paddle 2.6.1 and device Place(gpu:0)
[2024/06/06 03:38:02] ppocr INFO: Initialize indexs of datasets:['train_data/zzsfp/train.json']
list index out of range
(…)s/layoutxlm_base/sentencepiece.bpe.model: 100%|█████████████████████████████████████████████████████████████████| 5.07M/5.07M [00:00<00:00, 7.40MB/s]
[2024-06-06 03:38:10,232] [ INFO] - tokenizer config file saved in /root/.paddlenlp/models/layoutxlm-base-uncased/tokenizer_config.json
[2024-06-06 03:38:10,232] [ INFO] - Special tokens file saved in /root/.paddlenlp/models/layoutxlm-base-uncased/special_tokens_map.json
[2024/06/06 03:38:10] ppocr INFO: Initialize indexs of datasets:['train_data/zzsfp/val.json']
[2024-06-06 03:38:10,885] [ INFO] - tokenizer config file saved in /root/.paddlenlp/models/layoutxlm-base-uncased/tokenizer_config.json
[2024-06-06 03:38:10,885] [ INFO] - Special tokens file saved in /root/.paddlenlp/models/layoutxlm-base-uncased/special_tokens_map.json
[2024-06-06 03:38:10,891] [ WARNING] - You are using a model of type layoutlmv2 to instantiate a model of type layoutxlm. This is not supported for all configurations of models and can yield errors.
(…)outxlm-base-uncased/model_state.pdparams: 100%|█████████████████████████████████████████████████████████████████| 1.12G/1.12G [01:45<00:00, 10.6MB/s]
[2024-06-06 03:39:56,660] [ INFO] - Loading weights file from cache at /root/.paddlenlp/models/vi-layoutxlm-base-uncased/model_state.pdparams
[2024-06-06 03:39:57,666] [ INFO] - Loaded weights file from disk, setting weights to model.
W0606 03:39:57.742509 460 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.9, Driver API Version: 12.2, Runtime API Version: 12.0
W0606 03:39:57.743930 460 gpu_resources.cc:164] device: 0, cuDNN Version: 8.8.
[2024-06-06 03:39:59,033] [ WARNING] - Some weights of the model checkpoint at vi-layoutxlm-base-uncased were not used when initializing LayoutXLMForRelationExtraction: ['visual.pixel_mean', 'visual.pixel_std', 'visual_proj.bias', 'visual_proj.weight']

• This IS expected if you are initializing LayoutXLMForRelationExtraction from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing LayoutXLMForRelationExtraction from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  [2024-06-06 03:39:59,033] [ WARNING] - Some weights of LayoutXLMForRelationExtraction were not initialized from the model checkpoint at vi-layoutxlm-base-uncased and are newly initialized: ['extractor.ffnn_head.3.weight', 'extractor.rel_classifier.linear.weight', 'extractor.ffnn_tail.3.weight', 'extractor.ffnn_head.0.bias', 'extractor.ffnn_head.3.bias', 'extractor.rel_classifier.bilinear.weight', 'extractor.ffnn_tail.0.weight', 'extractor.entity_emb.weight', 'extractor.rel_classifier.linear.bias', 'extractor.ffnn_head.0.weight', 'extractor.ffnn_tail.3.bias', 'extractor.ffnn_tail.0.bias']
  You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  [2024-06-06 03:39:59,063] [ WARNING] - You are using a model of type layoutlmv2 to instantiate a model of type layoutxlm. This is not supported for all configurations of models and can yield errors.
  [2024-06-06 03:39:59,063] [ INFO] - Loading weights file from cache at /root/.paddlenlp/models/vi-layoutxlm-base-uncased/model_state.pdparams
  [2024-06-06 03:40:00,048] [ INFO] - Loaded weights file from disk, setting weights to model.
  [2024-06-06 03:40:01,242] [ WARNING] - Some weights of the model checkpoint at vi-layoutxlm-base-uncased were not used when initializing LayoutXLMForRelationExtraction: ['visual.pixel_mean', 'visual.pixel_std', 'visual_proj.bias', 'visual_proj.weight']
• This IS expected if you are initializing LayoutXLMForRelationExtraction from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing LayoutXLMForRelationExtraction from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  [2024-06-06 03:40:01,242] [ WARNING] - Some weights of LayoutXLMForRelationExtraction were not initialized from the model checkpoint at vi-layoutxlm-base-uncased and are newly initialized: ['extractor.ffnn_head.3.weight', 'extractor.rel_classifier.linear.weight', 'extractor.ffnn_tail.3.weight', 'extractor.ffnn_head.0.bias', 'extractor.ffnn_head.3.bias', 'extractor.rel_classifier.bilinear.weight', 'extractor.ffnn_tail.0.weight', 'extractor.entity_emb.weight', 'extractor.rel_classifier.linear.bias', 'extractor.ffnn_head.0.weight', 'extractor.ffnn_tail.3.bias', 'extractor.ffnn_tail.0.bias']
  You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  [2024/06/06 03:40:01] ppocr INFO: train dataloader has 15 iters
  [2024/06/06 03:40:01] ppocr INFO: valid dataloader has 1 iters
  [2024/06/06 03:40:01] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 19 iterations
  W0606 03:40:03.024000 460 gpu_resources.cc:299] WARNING: device: . The installed Paddle is compiled with CUDNN 8.9, but CUDNN version in your machine is 8.8, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
  ERROR: Unexpected BUS error encountered in DataLoader worker. This might be caused by insufficient shared memory (shm), please check whether use_shared_memory is set and storage space in /dev/shm is enough
  Traceback (most recent call last):
  File "/PaddleOCR/tools/train.py", line 255, in
  main(config, device, logger, vdl_writer, seed)
  File "/PaddleOCR/tools/train.py", line 208, in main
  program.train(
  File "/PaddleOCR/tools/program.py", line 339, in train
  preds = model(batch)
  File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call
  return self.forward(*inputs, **kwargs)
  File "/PaddleOCR/ppocr/modeling/architectures/distillation_model.py", line 59, in forward
  result_dict[model_name] = self.model_list[idx](x, data)
  File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call
  return self.forward(*inputs, **kwargs)
  File "/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 85, in forward
  x = self.backbone(x)
  File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call
  return self.forward(*inputs, **kwargs)
  File "/PaddleOCR/ppocr/modeling/backbones/vqa_layoutlm.py", line 248, in forward
  x = self.model(
  File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call
  return self.forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 1329, in forward
  loss, pred_relations = self.extractor(sequence_output, entities, relations)
  File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 1429, in call
  return self.forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 1223, in forward
  relations, entities = self.build_relation(relations, entities)
  File "/usr/local/lib/python3.10/dist-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 1189, in build_relation
  if negative_mask.sum() > 0:
  AttributeError: 'bool' object has no attribute 'sum'

[GreatV]

参考

• 按照教程将SER+RE串联执行，代码报错 argument 'x' (position 0) must be list of Tensors, but got empty list #12570