疑似梯度爆炸

[kyo-dai]

我是单卡训练，因为设备性能不足只能更改batchsize为16，但是训练结果显示，大约从20epoch开始损失一直这么大，没有减小的趋势，但我并没有改代码的其他内容，是什么原因，学习率过低了吗？
Epoch: [34/80] Total time: 1:01:24 (0.4984 s / it)
Averaged stats: lr: 0.000013 loss: 31.4573 (31.6999) loss_vfl: 0.6523 (0.6711) loss_bbox: 0.1718 (0.1814) loss_giou: 0.5420 (0.5580) loss_fgl: 0.9576 (0.9586) loss_vfl_aux_0: 0.6924 (0.7414) loss_bbox_aux_0: 0.1809 (0.1943) loss_giou_aux_0: 0.5732 (0.5833) loss_fgl_aux_0: 0.9859 (0.9925) loss_ddf_aux_0: 0.0465 (0.0524) loss_vfl_aux_1: 0.6758 (0.7233) loss_bbox_aux_1: 0.1727 (0.1843) loss_giou_aux_1: 0.5473 (0.5635) loss_fgl_aux_1: 0.9565 (0.9616) loss_ddf_aux_1: 0.0063 (0.0074) loss_vfl_aux_2: 0.6636 (0.7034) loss_bbox_aux_2: 0.1720 (0.1819) loss_giou_aux_2: 0.5437 (0.5587) loss_fgl_aux_2: 0.9569 (0.9588) loss_ddf_aux_2: 0.0011 (0.0012) loss_vfl_aux_3: 0.6641 (0.6820) loss_bbox_aux_3: 0.1719 (0.1815) loss_giou_aux_3: 0.5429 (0.5580) loss_fgl_aux_3: 0.9577 (0.9587) loss_ddf_aux_3: 0.0002 (0.0002) loss_vfl_aux_4: 0.6494 (0.6732) loss_bbox_aux_4: 0.1718 (0.1814) loss_giou_aux_4: 0.5420 (0.5580) loss_fgl_aux_4: 0.9576 (0.9586) loss_ddf_aux_4: 0.0001 (0.0001) loss_vfl_pre: 0.6992 (0.7439) loss_bbox_pre: 0.1814 (0.1945) loss_giou_pre: 0.5710 (0.5819) loss_vfl_enc_0: 0.7163 (0.7446) loss_bbox_enc_0: 0.2078 (0.2262) loss_giou_enc_0: 0.6412 (0.6537) loss_vfl_dn_0: 0.4922 (0.4909) loss_bbox_dn_0: 0.2619 (0.2536) loss_giou_dn_0: 0.5637 (0.5719) loss_fgl_dn_0: 1.0806 (1.0913) loss_ddf_dn_0: 0.2193 (0.2250) loss_vfl_dn_1: 0.4231 (0.4260) loss_bbox_dn_1: 0.1960 (0.1881) loss_giou_dn_1: 0.4617 (0.4596) loss_fgl_dn_1: 0.9825 (0.9864) loss_ddf_dn_1: 0.0265 (0.0283) loss_vfl_dn_2: 0.4050 (0.4107) loss_bbox_dn_2: 0.1827 (0.1780) loss_giou_dn_2: 0.4467 (0.4432) loss_fgl_dn_2: 0.9741 (0.9764) loss_ddf_dn_2: 0.0056 (0.0063) loss_vfl_dn_3: 0.3950 (0.4020) loss_bbox_dn_3: 0.1742 (0.1748) loss_giou_dn_3: 0.4411 (0.4389) loss_fgl_dn_3: 0.9761 (0.9772) loss_ddf_dn_3: 0.0007 (0.0009) loss_vfl_dn_4: 0.3960 (0.3984) loss_bbox_dn_4: 0.1685 (0.1740) loss_giou_dn_4: 0.4403 (0.4376) loss_fgl_dn_4: 0.9764 (0.9778) loss_ddf_dn_4: 0.0000 (0.0001) loss_vfl_dn_5: 0.3958 (0.3977) loss_bbox_dn_5: 0.1685 (0.1740) loss_giou_dn_5: 0.4404 (0.4376) loss_fgl_dn_5: 0.9765 (0.9778) loss_ddf_dn_5: 0.0000 (0.0000) loss_vfl_dn_pre: 0.4922 (0.4915) loss_bbox_dn_pre: 0.2697 (0.2594) loss_giou_dn_pre: 0.5670 (0.5709)
Test: [ 0/79] eta: 0:03:32 time: 2.6882 data: 1.8266 max mem: 17421
Test: [10/79] eta: 0:01:26 time: 1.2489 data: 0.2143 max mem: 17421
Test: [20/79] eta: 0:01:10 time: 1.1158 data: 0.0665 max mem: 17421
Test: [30/79] eta: 0:00:56 time: 1.1131 data: 0.0667 max mem: 17421
Test: [40/79] eta: 0:00:45 time: 1.1162 data: 0.0663 max mem: 17421
Test: [50/79] eta: 0:00:32 time: 1.0364 data: 0.0663 max mem: 17421
Test: [60/79] eta: 0:00:21 time: 1.0394 data: 0.0666 max mem: 17421
Test: [70/79] eta: 0:00:10 time: 1.1245 data: 0.0666 max mem: 17421
Test: [78/79] eta: 0:00:01 time: 1.0807 data: 0.0603 max mem: 17421
Test: Total time: 0:01:27 (1.1121 s / it)
Metrics: {'f1': 0.6701090231742727, 'precision': 0.6669847588406903, 'recall': 0.6732626943718178, 'iou': 0.4351902008130304, 'TPs': 24463, 'FPs': 12214, 'FNs': 11872}
Averaged stats:
Accumulating evaluation results...
COCOeval_opt.accumulate() finished...
DONE (t=3.74s).
IoU metric: bbox
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.527
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.704
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.572
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.338
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.570
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.712
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.390
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.654
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.721
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.539
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.768
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.892
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.916
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.785
best_stat: {'epoch': 34, 'coco_eval_bbox': 0.5272979428131327}