how to fix loss nan in FP16 training

[kellenf]

When I use FP16(pytorch amp API), I meet loss nan bug.
So how about your AMP code? Does it can fix loss_nan? Or do you meet loss_nan in amp training ?How do you fix this problem?

Thank you for your answer.

[C1rN09]

There are many reasons causing nan in FP16 training. You can refer to PyTorch documentation for advices.

MMEngine uses PyTorch autocast as AMP training strategy, so it typically behaves the same as native PyTorch. However, we handle advanced usages (e.g. gradient clipping, gradient accumulation) well, so this should prevent you from some issues mentioned in the above link.