rec 模型训练，训练突然变差的情况越来越频繁了

[nissansz]

问题描述 / Problem Description

运行环境 / Runtime Environment

• OS:
• Paddle: 2.3.2
• PaddleOCR: 2.7.5

复现代码 / Reproduction Code

完整报错 / Complete Error Message

可能解决方案 / Possible solutions

附件 / Appendix

[2024/06/12 20:01:07] ppocr INFO: epoch: [1/2], global_step: 9800, lr: 0.000500, acc: 0.499999, norm_edit_dis: 0.824553, loss: 11.132833, avg_reader_cost: 0.01647 s, avg_batch_cost: 1.00704 s, avg_samples: 8.0, ips: 7.94411 samples/s, eta: 22 days, 18:19:26
[2024/06/12 20:02:48] ppocr INFO: epoch: [1/2], global_step: 9900, lr: 0.000500, acc: 0.499999, norm_edit_dis: 0.821987, loss: 8.620522, avg_reader_cost: 0.04995 s, avg_batch_cost: 1.01023 s, avg_samples: 8.0, ips: 7.91898 samples/s, eta: 22 days, 17:31:18
[2024/06/12 20:04:25] ppocr INFO: epoch: [1/2], global_step: 10000, lr: 0.000500, acc: 0.375000, norm_edit_dis: 0.766948, loss: 15.263677, avg_reader_cost: 0.00143 s, avg_batch_cost: 0.96788 s, avg_samples: 8.0, ips: 8.26552 samples/s, eta: 22 days, 16:32:17
[2024/06/12 20:07:23] ppocr INFO: cur metric, acc: 0.49804687305450435, norm_edit_dis: 0.8151588771862497, fps: 67.39081284037641
[2024/06/12 20:07:24] ppocr INFO: save model in C:/Users/Administrator/Desktop/4lan5000fonts\latest
[2024/06/12 20:07:24] ppocr INFO: best metric, acc: 0.5882812477020263, is_float16: False, norm_edit_dis: 0.8538320936645203, fps: 59.28932826648075, best_epoch: 1
[2024/06/12 20:08:56] ppocr INFO: epoch: [1/2], global_step: 10100, lr: 0.000500, acc: 0.437499, norm_edit_dis: 0.835975, loss: 10.299061, avg_reader_cost: 0.04396 s, avg_batch_cost: 0.92275 s, avg_samples: 8.0, ips: 8.66977 samples/s, eta: 22 days, 15:21:56
[2024/06/12 20:10:33] ppocr INFO: epoch: [1/2], global_step: 10200, lr: 0.000500, acc: 0.375000, norm_edit_dis: 0.811421, loss: 13.557803, avg_reader_cost: 0.00910 s, avg_batch_cost: 0.96636 s, avg_samples: 8.0, ips: 8.27851 samples/s, eta: 22 days, 14:24:52
[2024/06/12 20:12:11] ppocr INFO: epoch: [1/2], global_step: 10300, lr: 0.000500, acc: 0.375000, norm_edit_dis: 0.798174, loss: 13.946710, avg_reader_cost: 0.00412 s, avg_batch_cost: 0.97799 s, avg_samples: 8.0, ips: 8.18007 samples/s, eta: 22 days, 13:32:01
[2024/06/12 20:13:48] ppocr INFO: epoch: [1/2], global_step: 10400, lr: 0.000500, acc: 0.499999, norm_edit_dis: 0.827978, loss: 11.830282, avg_reader_cost: 0.00162 s, avg_batch_cost: 0.97769 s, avg_samples: 8.0, ips: 8.18253 samples/s, eta: 22 days, 12:40:05
[2024/06/12 20:15:33] ppocr INFO: epoch: [1/2], global_step: 10500, lr: 0.000500, acc: 0.499999, norm_edit_dis: 0.843849, loss: 8.367547, avg_reader_cost: 0.00071 s, avg_batch_cost: 1.04856 s, avg_samples: 8.0, ips: 7.62951 samples/s, eta: 22 days, 12:07:56
[2024/06/12 20:17:16] ppocr INFO: epoch: [1/2], global_step: 10600, lr: 0.000500, acc: 0.125000, norm_edit_dis: 0.635646, loss: 27.141296, avg_reader_cost: 0.02415 s, avg_batch_cost: 1.02771 s, avg_samples: 8.0, ips: 7.78429 samples/s, eta: 22 days, 11:30:53
[2024/06/12 20:18:59] ppocr INFO: epoch: [1/2], global_step: 10700, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.719922, loss: 20.356087, avg_reader_cost: 0.01209 s, avg_batch_cost: 1.02881 s, avg_samples: 8.0, ips: 7.77597 samples/s, eta: 22 days, 10:54:46
[2024/06/12 20:20:40] ppocr INFO: epoch: [1/2], global_step: 10800, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.707201, loss: 24.053764, avg_reader_cost: 0.00139 s, avg_batch_cost: 1.01194 s, avg_samples: 8.0, ips: 7.90562 samples/s, eta: 22 days, 10:14:56
[2024/06/12 20:22:23] ppocr INFO: epoch: [1/2], global_step: 10900, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.710918, loss: 18.731030, avg_reader_cost: 0.00071 s, avg_batch_cost: 1.02554 s, avg_samples: 8.0, ips: 7.80080 samples/s, eta: 22 days, 9:39:17
[2024/06/12 20:24:06] ppocr INFO: epoch: [1/2], global_step: 11000, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.668536, loss: 18.735773, avg_reader_cost: 0.00241 s, avg_batch_cost: 1.02732 s, avg_samples: 8.0, ips: 7.78724 samples/s, eta: 22 days, 9:04:43
[2024/06/12 20:25:47] ppocr INFO: epoch: [1/2], global_step: 11100, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.689971, loss: 24.939096, avg_reader_cost: 0.00576 s, avg_batch_cost: 1.01268 s, avg_samples: 8.0, ips: 7.89981 samples/s, eta: 22 days, 8:27:02
[2024/06/12 20:27:26] ppocr INFO: epoch: [1/2], global_step: 11200, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.733670, loss: 15.895963, avg_reader_cost: 0.02417 s, avg_batch_cost: 0.99132 s, avg_samples: 8.0, ips: 8.07005 samples/s, eta: 22 days, 7:44:42
[2024/06/12 20:29:08] ppocr INFO: epoch: [1/2], global_step: 11300, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.736011, loss: 18.033504, avg_reader_cost: 0.00314 s, avg_batch_cost: 1.02255 s, avg_samples: 8.0, ips: 7.82357 samples/s, eta: 22 days, 7:10:47
[2024/06/12 20:30:51] ppocr INFO: epoch: [1/2], global_step: 11400, lr: 0.000500, acc: 0.375000, norm_edit_dis: 0.724694, loss: 16.869921, avg_reader_cost: 0.01272 s, avg_batch_cost: 1.02237 s, avg_samples: 8.0, ips: 7.82492 samples/s, eta: 22 days, 6:37:23
[2024/06/12 20:32:34] ppocr INFO: epoch: [1/2], global_step: 11500, lr: 0.000500, acc: 0.375000, norm_edit_dis: 0.783486, loss: 16.528812, avg_reader_cost: 0.02521 s, avg_batch_cost: 1.02952 s, avg_samples: 8.0, ips: 7.77064 samples/s, eta: 22 days, 6:06:16
[2024/06/12 20:34:12] ppocr INFO: epoch: [1/2], global_step: 11600, lr: 0.000500, acc: 0.375000, norm_edit_dis: 0.741096, loss: 17.632938, avg_reader_cost: 0.01827 s, avg_batch_cost: 0.98318 s, avg_samples: 8.0, ips: 8.13682 samples/s, eta: 22 days, 5:24:32
[2024/06/12 20:35:57] ppocr INFO: epoch: [1/2], global_step: 11700, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.747991, loss: 21.133224, avg_reader_cost: 0.00534 s, avg_batch_cost: 1.05049 s, avg_samples: 8.0, ips: 7.61549 samples/s, eta: 22 days, 4:59:31
[2024/06/12 20:37:53] ppocr INFO: epoch: [1/2], global_step: 11800, lr: 0.000500, acc: 0.312500, norm_edit_dis: 0.742141, loss: 16.772903, avg_reader_cost: 0.00938 s, avg_batch_cost: 1.16242 s, avg_samples: 8.0, ips: 6.88221 samples/s, eta: 22 days, 5:01:21
[2024/06/12 20:39:31] ppocr INFO: epoch: [1/2], global_step: 11900, lr: 0.000500, acc: 0.250000, norm_edit_dis: 0.719491, loss: 22.850651, avg_reader_cost: 0.01046 s, avg_batch_cost: 0.97884 s, avg_samples: 8.0, ips: 8.17291 samples/s, eta: 22 days, 4:20:06