Training Log 2022-11-22

[zh-zheng]

CPM-Live Training Log (November, 22)

Time: November, 22 2022 19:00

Recorder: @zh-zheng

Loss

• Begin: 2.21
• End: 2.17

Completed Data

• $\approx$ 1798.23GB

Average Grad Norm

• 1.22

Progress

• 40.62%

Comment

Today, the training loss became NaN at around 12:00. We solved this problem by scaling weights. We'll keep an eye on the model in the next few days.

[Aspirinkb]

可以详细介绍一下如何应用scaling weights吗？具体怎么做的？为什么要用这种方法解决NaN问题呢？谢谢，提前。