关于PP-OCRv3训练速度问题

[huilunc]

如题，目前在训练检测模型，648张图，batch size 8，500个epoch，在colab上用t4跑 大概需要4天，这个速度正常吗？

[2024/05/31 10:46:47] ppocr INFO: epoch: [1/500], global_step: 10, lr: 0.000030, dila_dbloss_Student_Teacher: 1.203395, dila_dbloss_Student2_Teacher: 1.203395, loss: 15.206206, dml_thrink_maps_0: 0.000000, db_Student_loss_shrink_maps: 4.393258, db_Student_loss_threshold_maps: 1.121766, db_Student_loss_binary_maps: 0.878726, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.393258, db_Student2_loss_threshold_maps: 1.121766, db_Student2_loss_binary_maps: 0.878726, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 9.62344 s, avg_batch_cost: 11.65222 s, avg_samples: 8.0, ips: 0.68656 samples/s, eta: 4 days, 23:43:35, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB
[2024/05/31 10:48:42] ppocr INFO: epoch: [1/500], global_step: 20, lr: 0.000064, dila_dbloss_Student_Teacher: 1.163067, dila_dbloss_Student2_Teacher: 1.163066, loss: 15.048415, dml_thrink_maps_0: 0.000000, db_Student_loss_shrink_maps: 4.460459, db_Student_loss_threshold_maps: 1.092649, db_Student_loss_binary_maps: 0.892236, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.460459, db_Student2_loss_threshold_maps: 1.092649, db_Student2_loss_binary_maps: 0.892236, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 9.58579 s, avg_batch_cost: 11.42676 s, avg_samples: 8.0, ips: 0.70011 samples/s, eta: 4 days, 22:32:10, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB
[2024/05/31 10:50:34] ppocr INFO: epoch: [1/500], global_step: 30, lr: 0.000132, dila_dbloss_Student_Teacher: 0.916454, dila_dbloss_Student2_Teacher: 0.916452, loss: 14.598045, dml_thrink_maps_0: 0.000000, db_Student_loss_shrink_maps: 4.473489, db_Student_loss_threshold_maps: 0.960112, db_Student_loss_binary_maps: 0.894671, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.473488, db_Student2_loss_threshold_maps: 0.960111, db_Student2_loss_binary_maps: 0.894671, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 9.40157 s, avg_batch_cost: 11.24571 s, avg_samples: 8.0, ips: 0.71138 samples/s, eta: 4 days, 21:29:54, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB
[2024/05/31 10:52:16] ppocr INFO: epoch: [1/500], global_step: 40, lr: 0.000199, dila_dbloss_Student_Teacher: 0.791128, dila_dbloss_Student2_Teacher: 0.791126, loss: 13.792257, dml_thrink_maps_0: 0.000001, db_Student_loss_shrink_maps: 4.365004, db_Student_loss_threshold_maps: 0.893057, db_Student_loss_binary_maps: 0.873257, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.365006, db_Student2_loss_threshold_maps: 0.893056, db_Student2_loss_binary_maps: 0.873258, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 8.28134 s, avg_batch_cost: 10.15662 s, avg_samples: 8.0, ips: 0.78766 samples/s, eta: 4 days, 18:10:07, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB
[2024/05/31 10:53:58] ppocr INFO: epoch: [1/500], global_step: 50, lr: 0.000267, dila_dbloss_Student_Teacher: 0.716208, dila_dbloss_Student2_Teacher: 0.716216, loss: 13.479249, dml_thrink_maps_0: 0.000001, db_Student_loss_shrink_maps: 4.398167, db_Student_loss_threshold_maps: 0.841628, db_Student_loss_binary_maps: 0.879189, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.398158, db_Student2_loss_threshold_maps: 0.841628, db_Student2_loss_binary_maps: 0.879187, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 8.34913 s, avg_batch_cost: 10.22874 s, avg_samples: 8.0, ips: 0.78211 samples/s, eta: 4 days, 16:18:27, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB
[2024/05/31 10:55:44] ppocr INFO: epoch: [1/500], global_step: 60, lr: 0.000334, dila_dbloss_Student_Teacher: 0.668552, dila_dbloss_Student2_Teacher: 0.668559, loss: 13.441912, dml_thrink_maps_0: 0.000001, db_Student_loss_shrink_maps: 4.380462, db_Student_loss_threshold_maps: 0.816749, db_Student_loss_binary_maps: 0.875890, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.380451, db_Student2_loss_threshold_maps: 0.816751, db_Student2_loss_binary_maps: 0.875887, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 8.68966 s, avg_batch_cost: 10.55330 s, avg_samples: 8.0, ips: 0.75806 samples/s, eta: 4 days, 15:36:44, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB
[2024/05/31 10:57:32] ppocr INFO: epoch: [1/500], global_step: 70, lr: 0.000402, dila_dbloss_Student_Teacher: 0.640622, dila_dbloss_Student2_Teacher: 0.640583, loss: 13.169693, dml_thrink_maps_0: 0.000001, db_Student_loss_shrink_maps: 4.317860, db_Student_loss_threshold_maps: 0.775226, db_Student_loss_binary_maps: 0.863769, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.317872, db_Student2_loss_threshold_maps: 0.775228, db_Student2_loss_binary_maps: 0.863768, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 8.98417 s, avg_batch_cost: 10.84738 s, avg_samples: 8.0, ips: 0.73750 samples/s, eta: 4 days, 15:32:18, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB
[2024/05/31 10:58:15] ppocr INFO: epoch: [1/500], global_step: 74, lr: 0.000429, dila_dbloss_Student_Teacher: 0.627995, dila_dbloss_Student2_Teacher: 0.628024, loss: 13.142283, dml_thrink_maps_0: 0.000001, db_Student_loss_shrink_maps: 4.330738, db_Student_loss_threshold_maps: 0.766503, db_Student_loss_binary_maps: 0.866224, db_Student_loss_cbn: 0.000000, db_Student2_loss_shrink_maps: 4.330718, db_Student2_loss_threshold_maps: 0.766506, db_Student2_loss_binary_maps: 0.866217, db_Student2_loss_cbn: 0.000000, avg_reader_cost: 3.56069 s, avg_batch_cost: 4.29096 s, avg_samples: 3.2, ips: 0.74575 samples/s, eta: 4 days, 15:26:44, max_mem_reserved: 14162 MB, max_mem_allocated: 13272 MB

[GreatV]

正常