如何使用混合精度训练文本检测模型？

[Simon-Guan]

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

• 系统环境/System Environment：ubuntu
• 版本号/Version：Paddle：2.3.2 PaddleOCR： 问题相关组件/Related components：
• 运行指令/Command Code：python tools/train.py -c configs/det/det_res18_db_v2.0.yml
• 完整报错/Complete Error Message：
  ....
  [2024/02/27 01:15:04] ppocr INFO: cur metric, precision: 0.0, recall: 0.0, hmean: 0, fps: 26.835780432923727
  [2024/02/27 01:15:10] ppocr INFO: save best model is to ./output/det_tk_20240226_ampO2/best_accuracy
  [2024/02/27 01:15:10] ppocr INFO: best metric, hmean: 0, is_float16: False, precision: 0.0, recall: 0.0, fps: 26.835780432923727, best_epoch: 8
  [2024/02/27 01:15:13] ppocr INFO: save model in ./output/det_tk_20240226_ampO2/latest
  [2024/02/27 01:15:16] ppocr INFO: save model in ./output/det_tk_20240226_ampO2/iter_epoch_8
  Found inf or nan, current scale is: 0.0, decrease to: 0.00.5
  [2024/02/27 01:15:36] ppocr INFO: epoch: [9/100], global_step: 6114, lr: 0.000991, loss: 9.693674, loss_shrink_maps: 4.770591, loss_threshold_maps: 3.967761, loss_binary_maps: 0.956001, loss_cbn: 0.000000, avg_reader_cost: 9.73265 s, avg_batch_cost: 13.14799 s, avg_samples: 32.0, ips: 2.43383 samples/s, eta: 1 day, 22:38:41
  Found inf or nan, current scale is: 0.0, decrease to: 0.00.5
  [2024/02/27 01:15:46] ppocr INFO: epoch: [9/100], global_step: 6116, lr: 0.000991, loss: 9.693674, loss_shrink_maps: 4.770591, loss_threshold_maps: 3.976369, loss_binary_maps: 0.956001, loss_cbn: 0.000000, avg_reader_cost: 0.00044 s, avg_batch_cost: 4.70846 s, avg_samples: 32.0, ips: 6.79628 samples/s, eta: 1 day, 22:39:29
  Found inf or nan, current scale is: 0.0, decrease to: 0.00.5
  [2024/02/27 01:15:49] ppocr INFO: epoch: [9/100], global_step: 6118, lr: 0.000991, loss: 9.691466, loss_shrink_maps: 4.757706, loss_threshold_maps: 3.976369, loss_binary_maps: 0.953352, loss_cbn: 0.000000, avg_reader_cost: 0.00045 s, avg_batch_cost: 1.33806 s, avg_samples: 32.0, ips: 23.91525 samples/s, eta: 1 day, 22:39:00
  Found inf or nan, current scale is: 0.0, decrease to: 0.00.5
  .......
  Global:
  use_gpu: true
  use_amp: true
  amp_level: O2
  amp_dtype: float16
  amp_custom_black_list: []
  amp_custom_white_list: []
  scale_loss: 1024
  use_dynamic_loss_scaling: true
  epoch_num: 100
  log_smooth_window: 20
  .....
  一、如何使用混合精度训练文本检测模型？
  二、设置使用amp模式，训练精度为0，请问检测模型的配置文件该如何修改？

我们提供了AceIssueSolver来帮助你解答问题，你是否想要它来解答(请填写yes/no)?/We provide AceIssueSolver to solve issues, do you want it? (Please write yes/no):yes

请尽量不要包含图片在问题中/Please try to not include the image in the issue.

[terrencew]

Same problem

[TingquanGao]

建议尝试：amp_level: O1

[Simon-Guan]

O1和O2都尝试过，精度一直很低，可以忽略掉，不采用混合精度训练就不会出现这个问题，请问你是怎么解决这个问题？ 发自我的 iPhone 在 2024年2月28日，12:03，Tingquan Gao ***@***.***> 写道：  建议尝试：amp_level: O1 — Reply to this email directly, view it on GitHub<#11626 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BEGKRQ2O5YA6HFX6UVTQTODYV2UBRAVCNFSM6AAAAABD3H3JKCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRYGE3TMMJVGE>. You are receiving this because you authored the thread.Message ID: ***@***.***>