Training Log 2022-06-16

[zh-zheng]

CPM-Live Training Log (June, 16)

Time: June, 16 2022 16:00

Recorder: @zh-zheng

Loss

• Begin: 2.58
• End: 2.47

Completed Data

• $\approx$ 162.17GB

Average Grad Norm

• 0.77

Progress

• 17.41%

Comment

Around 8:30 this morning, one of the nodes in our computing cluster lost its GPUs. We restarted that node and resumed the training at 12:10.