[Training Log 2022-10-15] We reduced the learning rate twice today.

[zh-zheng]

CPM-Live Training Log (October, 15)

Time: October, 15 2022 19:00

Recorder: @zh-zheng

Loss

• Begin: 10.75
• End: 2.44

Completed Data

• $\approx$ 1,260.09GB

Average Grad Norm

• 26.26

Progress

• 2.0%

Comment

Today's key events:

• The training loss increased rapidly at 5:00 and eventually became NaN. We reduced the learning rate by half and resumed the training at 10:40.

• We observed that the loss increased again at 16:40, so we reduced the learning rate by a factor of 0.2.

We'll keep an eye on the training process.