[Training Log 2022-10-23] We modify the computation of attention scores to stabilize the training.

[zh-zheng]

CPM-Live Training Log (October, 23)

Time: October, 23 2022 19:00

Recorder: @zh-zheng

Loss

• Begin: 2.51
• End: 2.48

Completed Data

• $\approx$ 1,354.55GB

Average Grad Norm

• 0.62

Progress

• 8.74%

Comment

We observed that the training loss became NaN this morning, and we stabilized the training by modifying the computation of attention scores. It seems to be working so far.