kie_ser训练问题

[gomatthew]

问题描述 / Problem Description

进行kie_ser训练时报下面的错,eval的数据和 train数据格式是相同的,请大佬看看

运行环境 / Runtime Environment

• PaddleOCR:

复现代码 / Reproduction Code

完整报错 / Complete Error Message

可能解决方案 / Possible solutions

附件 / Appendix

[SWHL]

需要提供一个最小可复现的demo，才好排查问题，从截图猜不出来是啥原因

[gomatthew]

这是 完整的训练配置文件:
Global:
use_gpu: True
epoch_num: &epoch_num 200
log_smooth_window: 10
print_batch_step: 10
save_model_dir: ./output/ser_vi_layoutxlm_xfund_zh_udml
save_epoch_step: 2000

evaluation is run every 10 iterations after the 0th iteration

eval_batch_step: [ 0, 19 ]
cal_metric_during_train: False
save_inference_dir:
use_visualdl: False
seed: 2022
infer_img: ppstructure/docs/kie/input/zh_val_42.jpg
save_res_path: ./output/ser_layoutxlm_xfund_zh/res

Architecture:
model_type: &model_type "kie"
name: DistillationModel
algorithm: Distillation
Models:
Teacher:
pretrained:
freeze_params: false
return_all_feats: true
model_type: *model_type
algorithm: &algorithm "LayoutXLM"
Transform:
Backbone:
name: LayoutXLMForSer
pretrained: True
# one of base or vi
mode: vi
checkpoints:
num_classes: &num_classes 200
Student:
pretrained:
freeze_params: false
return_all_feats: true
model_type: *model_type
algorithm: *algorithm
Transform:
Backbone:
name: LayoutXLMForSer
pretrained: True
# one of base or vi
mode: vi
checkpoints:
num_classes: *num_classes

Loss:
name: CombinedLoss
loss_config_list:

• DistillationVQASerTokenLayoutLMLoss:
  weight: 1.0
  model_name_list: ["Student", "Teacher"]
  key: backbone_out
  num_classes: *num_classes
• DistillationSERDMLLoss:
  weight: 1.0
  act: "softmax"
  use_log: true
  model_name_pairs:
  • ["Student", "Teacher"]
    key: backbone_out
• DistillationVQADistanceLoss:
  weight: 0.5
  mode: "l2"
  model_name_pairs:
  - ["Student", "Teacher"]
  key: hidden_states_5
  name: "loss_5"
• DistillationVQADistanceLoss:
  weight: 0.5
  mode: "l2"
  model_name_pairs:
  - ["Student", "Teacher"]
  key: hidden_states_8
  name: "loss_8"

Optimizer:
name: AdamW
beta1: 0.9
beta2: 0.999
lr:
name: Linear
learning_rate: 0.0005
epochs: *epoch_num
warmup_epoch: 10
regularizer:
name: L2
factor: 0.00000

PostProcess:
name: DistillationSerPostProcess
model_name: ["Student", "Teacher"]
key: backbone_out
class_path: &class_path train_data/train/class_list.txt

Metric:
name: DistillationMetric
base_metric_name: VQASerTokenMetric
main_indicator: hmean
key: "Student"

Train:
dataset:
name: SimpleDataSet
data_dir: train_data/train/imgs
label_file_list:
- train_data/train/train.json
ratio_list: [ 1.0 ]
transforms:
- DecodeImage: # load image
img_mode: RGB
channel_first: False
- VQATokenLabelEncode: # Class handling label
contains_re: False
algorithm: *algorithm
class_path: *class_path
# one of [None, "tb-yx"]
order_method: &order_method "tb-yx"
- VQATokenPad:
max_seq_len: &max_seq_len 512
return_attention_mask: True
- VQASerTokenChunk:
max_seq_len: *max_seq_len
- Resize:
size: [224,224]
- NormalizeImage:
scale: 1
mean: [ 123.675, 116.28, 103.53 ]
std: [ 58.395, 57.12, 57.375 ]
order: 'hwc'
- ToCHWImage:
- KeepKeys:
keep_keys: [ 'input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'image', 'labels'] # dataloader will return list in this order
loader:
shuffle: True
drop_last: False
batch_size_per_card: 4
num_workers: 0

Eval:
dataset:
name: SimpleDataSet
data_dir: train_data/train/imgs
label_file_list:
- train_data/train/val.json
transforms:
- DecodeImage: # load image
img_mode: RGB
channel_first: False
- VQATokenLabelEncode: # Class handling label
contains_re: False
algorithm: *algorithm
class_path: *class_path
order_method: *order_method
- VQATokenPad:
max_seq_len: *max_seq_len
return_attention_mask: True
- VQASerTokenChunk:
max_seq_len: *max_seq_len
- Resize:
size: [224,224]
- NormalizeImage:
scale: 1
mean: [ 123.675, 116.28, 103.53 ]
std: [ 58.395, 57.12, 57.375 ]
order: 'hwc'
- ToCHWImage:
- KeepKeys:
keep_keys: [ 'input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'image', 'labels'] # dataloader will return list in this order
loader:
shuffle: False
drop_last: False
batch_size_per_card: 8
num_workers: 0

这是训练报错的数据集 :
train_data.json
1.jpg [{"id": 0, "transcription": "034002300104", "points": [[400, 149], [747, 149], [747, 200], [400, 200]], "label": "receipt_code"}, {"id": 1, "transcription": "开票日期", "points": [[1393, 255], [1512, 255], [1512, 295], [1393, 295]], "label": "q_receipt_date", "linking": [[1, 2]]}, {"id": 2, "transcription": "2023年12月24日", "points": [[1533, 246], [1730, 246], [1730, 279], [1533, 279]], "label": "a_receipt_date", "linking": [[1, 2]]}, {"id": 3, "transcription": "名称", "points": [[280, 309], [452, 309], [452, 348], [280, 348]], "label": "q_buy_name", "linking": [[3, 4]]}, {"id": 4, "transcription": "惠州华阳通用电子有限公司", "points": [[495, 293], [810, 293], [810, 334], [495, 334]], "label": "a_buy_name", "linking": [[3, 4]]}, {"id": 5, "transcription": "纳税人识别号", "points": [[280, 353], [452, 353], [452, 391], [280, 391]], "label": "q_buy_person_id", "linking": [[5, 6]]}, {"id": 6, "transcription": "91441300743666938D", "points": [[489, 334], [870, 334], [870, 376], [489, 376]], "label": "a_buy_person_id", "linking": [[5, 6]]}, {"id": 7, "transcription": "地址电话", "points": [[280, 397], [451, 397], [451, 428], [280, 428]], "label": "q_buy_addr_tel", "linking": [[7, 8]]}, {"id": 8, "transcription": "广东省州市东江高新科技产业园上霞北路1号华阳工业园A区2号0752-2618616", "points": [[493, 372], [1124, 372], [1124, 425], [493, 425]], "label": "a_buy_addr_tel", "linking": [[7, 8]]}, {"id": 9, "transcription": "开户行及账号", "points": [[282, 435], [452, 435], [452, 476], [282, 476]], "label": "q_buy_bank", "linking": [[9, 10]]}, {"id": 10, "transcription": "中国农业银行股份有限公司惠州下埔支行44224201040001269", "points": [[493, 427], [1098, 427], [1098, 462], [493, 462]], "label": "a_buy_bank", "linking": [[9, 10]]}, {"id": 11, "transcription": "货物或应税劳务、服务名称", "points": [[251, 483], [591, 483], [591, 520], [251, 520]], "label": "q_buy_good", "linking": [[11, 12]]}, {"id": 12, "transcription": "住宿服务住宿服务", "points": [[235, 498], [484, 498], [484, 535], [235, 535]], "label": "a_buy_good", "linking": [[11, 12]]}, {"id": 13, "transcription": "单位", "points": [[858, 488], [914, 488], [914, 518], [858, 518]], "label": "q_unit", "linking": [[13, 14]]}, {"id": 14, "transcription": "天", "points": [[852, 504], [884, 504], [884, 534], [852, 534]], "label": "a_unit", "linking": [[13, 14]]}, {"id": 15, "transcription": "数量", "points": [[977, 488], [1052, 488], [1052, 523], [977, 523]], "label": "q_amount", "linking": [[15, 16]]}, {"id": 16, "transcription": "3", "points": [[1079, 506], [1096, 506], [1096, 532], [1079, 532]], "label": "a_amount", "linking": [[15, 16]]}, {"id": 17, "transcription": "单价", "points": [[1133, 488], [1212, 488], [1212, 518], [1133, 518]], "label": "q_unit_price", "linking": [[17, 18]]}, {"id": 18, "transcription": "156.435643564", "points": [[1117, 507], [1251, 507], [1251, 532], [1117, 532]], "label": "a_unit_price", "linking": [[17, 18]]}, {"id": 19, "transcription": "金额", "points": [[1316, 488], [1421, 488], [1421, 520], [1316, 520]], "label": "q_price", "linking": [[19, 20]]}, {"id": 20, "transcription": "￥469.31", "points": [[1370, 776], [1496, 776], [1496, 816], [1370, 816]], "label": "a_price", "linking": [[19, 20]]}, {"id": 21, "transcription": "税率", "points": [[1507, 491], [1563, 491], [1563, 518], [1507, 518]], "label": "q_tax_rate", "linking": [[21, 22]]}, {"id": 22, "transcription": "1%", "points": [[1526, 507], [1558, 507], [1558, 537], [1526, 537]], "label": "a_tax_rate", "linking": [[21, 22]]}, {"id": 23, "transcription": "税额", "points": [[1630, 488], [1737, 488], [1737, 521], [1630, 521]], "label": "q_tax", "linking": [[23, 24]]}, {"id": 24, "transcription": "￥4.69", "points": [[1710, 779], [1805, 779], [1805, 813], [1710, 813]], "label": "a_tax", "linking": [[23, 24]]}, {"id": 25, "transcription": "价税合计（大写）", "points": [[302, 846], [535, 846], [535, 886], [302, 886]], "label": "q_total_price_capital", "linking": [[25, 26]]}, {"id": 26, "transcription": "肆佰柒拾肆圆整", "points": [[680, 832], [895, 832], [895, 867], [680, 867]], "label": "a_total_price_capital", "linking": [[25, 26]]}, {"id": 27, "transcription": "小写", "points": [[1386, 846], [1440, 846], [1440, 881], [1386, 881]], "label": "q_total_price", "linking": [[27, 28]]}, {"id": 28, "transcription": "￥474.00", "points": [[1496, 834], [1631, 834], [1631, 865], [1496, 865]], "label": "a_total_price", "linking": [[27, 28]]}, {"id": 29, "transcription": "备注", "points": [[1142, 916], [1170, 916], [1170, 1032], [1142, 1032]], "label": "q_tip", "linking": [[29, 30]]}, {"id": 30, "transcription": "校验码01019 77757 14184 15016", "points": [[1209, 895], [1609, 895], [1609, 921], [1209, 921]], "label": "a_tip", "linking": [[29, 30]]}, {"id": 31, "transcription": "名称", "points": [[280, 907], [447, 907], [447, 942], [280, 942]], "label": "q_sale_name", "linking": [[31, 32]]}, {"id": 32, "transcription": "芜湖华鼎商务酒店有限责任公司", "points": [[491, 897], [866, 897], [866, 923], [491, 923]], "label": "a_sale_name", "linking": [[31, 32]]}, {"id": 33, "transcription": "纳税人识别号", "points": [[277, 949], [447, 949], [447, 977], [277, 977]], "label": "q_sale_person_id", "linking": [[33, 34]]}, {"id": 34, "transcription": "91340200059701944H", "points": [[493, 932], [866, 932], [866, 962], [493, 962]], "label": "a_sale_person_id", "linking": [[33, 34]]}, {"id": 35, "transcription": "地址电话", "points": [[279, 984], [445, 984], [445, 1013], [279, 1013]], "label": "q_sale_addr_tel", "linking": [[35, 36]]}, {"id": 36, "transcription": "芜湖市代江区峨山路与中山南路交叉口0553-5656777", "points": [[491, 976], [1100, 976], [1100, 1002], [491, 1002]], "label": "a_sale_addr_tel", "linking": [[35, 36]]}, {"id": 37, "transcription": "开户行及账号", "points": [[279, 1020], [449, 1020], [449, 1053], [279, 1053]], "label": "q_sale_bank", "linking": [[37, 38]]}, {"id": 38, "transcription": "芜湖杨子银行农村商业银行鲁港支行20000341467610300000018", "points": [[495, 1016], [1119, 1016], [1119, 1042], [495, 1042]], "label": "a_sale_bank", "linking": [[37, 38]]}, {"id": 39, "transcription": "22717855", "points": [[1413, 160], [1595, 160], [1595, 209], [1413, 209]], "label": "receipt_number"}]

评估数据集:
val.json
b26.jpg [{"id": 0, "transcription": "4400154130", "points": [[618, 471], [1289, 471], [1289, 590], [618, 590]], "label": "receipt_code"}, {"id": 1, "transcription": "12270798", "points": [[3004, 423], [3585, 423], [3585, 571], [3004, 571]], "label": "receipt_number"}, {"id": 2, "transcription": "开票日期", "points": [[2956, 666], [3275, 666], [3275, 761], [2956, 761]], "label": "q_receipt_date", "linking": [[2, 3]]}, {"id": 3, "transcription": "2016年06月12日", "points": [[3380, 647], [3875, 647], [3875, 728], [3380, 728]], "label": "a_receipt_date", "linking": [[2, 3]]}, {"id": 4, "transcription": "名称", "points": [[327, 852], [742, 852], [742, 938], [327, 938]], "label": "q_buy_name", "linking": [[4, 5]]}, {"id": 5, "transcription": "深圳市购机汇网络有限公司", "points": [[832, 847], [1647, 847], [1647, 933], [832, 933]], "label": "a_buy_name", "linking": [[4, 5]]}, {"id": 6, "transcription": "纳税人识别号", "points": [[323, 952], [708, 952], [708, 1019], [323, 1019]], "label": "q_buy_person_id", "linking": [[6, 7]]}, {"id": 7, "transcription": "440300083885931", "points": [[894, 938], [1742, 938], [1742, 1033], [894, 1033]], "label": "a_buy_person_id", "linking": [[6, 7]]}, {"id": 8, "transcription": "地址电话", "points": [[304, 1033], [723, 1033], [723, 1133], [304, 1133]], "label": "q_buy_addr_tel", "linking": [[8, 9]]}, {"id": 9, "transcription": "深圳市龙华新区民治街道民治大道展港科技大厦A12070755-23806606", "points": [[842, 1033], [2275, 1033], [2275, 1123], [842, 1123]], "label": "a_buy_addr_tel", "linking": [[8, 9]]}, {"id": 10, "transcription": "开户行及账号", "points": [[313, 1133], [727, 1133], [727, 1228], [313, 1228]], "label": "q_buy_bank", "linking": [[10, 11]]}, {"id": 11, "transcription": "中国工商银行股份有限公司深圳园岭支行4000024709200172809", "points": [[851, 1138], [2356, 1138], [2356, 1223], [851, 1223]], "label": "a_buy_bank", "linking": [[10, 11]]}, {"id": 12, "transcription": "货物或应税劳务、服务名称", "points": [[251, 1261], [1099, 1261], [1099, 1333], [251, 1333]], "label": "q_buy_good", "linking": [[12, 13]]}, {"id": 13, "transcription": "小通电信全国版时色", "points": [[218, 1352], [1123, 1352], [1123, 1514], [218, 1514]], "label": "a_buy_good", "linking": [[12, 13]]}, {"id": 14, "transcription": "单位", "points": [[1694, 1247], [1856, 1247], [1856, 1323], [1694, 1323]], "label": "q_unit", "linking": [[14, 15]]}, {"id": 15, "transcription": "个", "points": [[1747, 1333], [1847, 1333], [1847, 1423], [1747, 1423]], "label": "a_unit", "linking": [[14, 15]]}, {"id": 16, "transcription": "数量", "points": [[1966, 1242], [2199, 1242], [2199, 1333], [1966, 1333]], "label": "q_amount", "linking": [[16, 17]]}, {"id": 17, "transcription": "5", "points": [[2204, 1323], [2280, 1323], [2280, 1409], [2204, 1409]], "label": "a_amount", "linking": [[16, 17]]}, {"id": 18, "transcription": "单价", "points": [[2361, 1233], [2566, 1233], [2566, 1314], [2361, 1314]], "label": "q_unit_price", "linking": [[18, 19]]}, {"id": 19, "transcription": "597.43589744", "points": [[2332, 1328], [2685, 1328], [2685, 1414], [2332, 1414]], "label": "a_unit_price", "linking": [[18, 19]]}, {"id": 20, "transcription": "金额", "points": [[2799, 1228], [3089, 1228], [3089, 1304], [2799, 1304]], "label": "q_price", "linking": [[20, 21]]}, {"id": 21, "transcription": "¥2987.18", "points": [[2866, 1938], [3280, 1938], [3280, 2047], [2866, 2047]], "label": "a_price", "linking": [[20, 21]]}, {"id": 22, "transcription": "税率", "points": [[3251, 1214], [3404, 1214], [3404, 1300], [3251, 1300]], "label": "q_tax_rate", "linking": [[22, 23]]}, {"id": 23, "transcription": "17%", "points": [[3318, 1309], [3432, 1309], [3432, 1390], [3318, 1390]], "label": "a_tax_rate", "linking": [[22, 23]]}, {"id": 24, "transcription": "税额", "points": [[3570, 1209], [3856, 1209], [3856, 1300], [3570, 1300]], "label": "q_tax", "linking": [[24, 25]]}, {"id": 25, "transcription": "¥507.82", "points": [[3680, 1919], [4027, 1919], [4027, 2038], [3680, 2038]], "label": "a_tax", "linking": [[24, 25]]}, {"id": 26, "transcription": "价税合计（大写）", "points": [[323, 2138], [937, 2138], [937, 2233], [323, 2233]], "label": "q_total_price_capital", "linking": [[26, 27]]}, {"id": 27, "transcription": "叁仟肆佰玖拾伍圆整", "points": [[1266, 2104], [2032, 2104], [2032, 2200], [1266, 2200]], "label": "a_total_price_capital", "linking": [[26, 27]]}, {"id": 28, "transcription": "小写", "points": [[3018, 2080], [3156, 2080], [3156, 2176], [3018, 2176]], "label": "q_total_price", "linking": [[28, 29]]}, {"id": 29, "transcription": "￥3495.00", "points": [[3266, 2071], [3727, 2071], [3727, 2171], [3266, 2171]], "label": "a_total_price", "linking": [[28, 29]]}, {"id": 30, "transcription": "名称", "points": [[280, 2295], [713, 2295], [713, 2385], [280, 2385]], "label": "q_sale_name", "linking": [[30, 31]]}, {"id": 31, "transcription": "广州晶东贸易有限公司", "points": [[808, 2276], [1513, 2276], [1513, 2357], [808, 2357]], "label": "a_sale_name", "linking": [[30, 31]]}, {"id": 32, "transcription": "纳税人识别号", "points": [[270, 2380], [694, 2380], [694, 2471], [270, 2471]], "label": "q_sale_person_id", "linking": [[32, 33]]}, {"id": 33, "transcription": "91440101664041243T9", "points": [[894, 2371], [1989, 2371], [1989, 2457], [894, 2457]], "label": "a_sale_person_id", "linking": [[32, 33]]}, {"id": 34, "transcription": "地址电话", "points": [[289, 2476], [689, 2476], [689, 2571], [289, 2571]], "label": "q_sale_addr_tel", "linking": [[34, 35]]}, {"id": 35, "transcription": "广州市黄埔区九龙镇九龙工业园凤凰三横路99号88216600", "points": [[804, 2461], [2318, 2461], [2318, 2557], [804, 2557]], "label": "a_sale_addr_tel", "linking": [[34, 35]]}, {"id": 36, "transcription": "开户行及账号", "points": [[270, 2566], [680, 2566], [680, 2657], [270, 2657]], "label": "q_sale_bank", "linking": [[36, 37]]}, {"id": 37, "transcription": "工行北京路支行3602000919200384952", "points": [[813, 2557], [2023, 2557], [2023, 2633], [813, 2633]], "label": "a_sale_bank", "linking": [[36, 37]]}, {"id": 38, "transcription": "备注", "points": [[2399, 2271], [2504, 2271], [2504, 2580], [2399, 2580]], "label": "q_tip", "linking": [[38, 39]]}, {"id": 39, "transcription": "dd42981457299（00001,19609966o", "points": [[2527, 2233], [3589, 2233], [3589, 2319], [2527, 2319]], "label": "a_tip", "linking": [[38, 39]]}]

相应图片也保存在配置文件路径

训练命令为:
python tool/train.py -c path/ser_config.py

请大佬看看