Given the correct parameters and initial data set, why does NaN occur?

[12jscvb]

I used deepmd v2.2.9, dpgen v 0.12.0, but NaN always appeared in lcurve.out of the four initial potential function folders. May I ask why? I checked the initial data set and found it intact and no abnormal conditions. This problem has been bothering me for a long time and I couldn't solve it, so I asked for help.
This also happened when I used ·deepmd v2.2.8 before.
(The attachments are lcurve.out, train.log)
If you could give me any help, I'd appreciate it . Thank you

lcurve.txt

train.log

[njzjz]

It can't be ruled out that your gpu is broken. See https://forums.developer.nvidia.com/t/ubuntu-18-04-rtx-2080-ti-tensorflow-nan-values-consistently-appearing-during-training-of-all-networks/125869.
Perhaps use another machine and see if the same problem happens again.