Training Log 2022-06-19

[zh-zheng]

CPM-Live Training Log (June, 19)

Time: June, 19 2022 16:00

Recorder: @zh-zheng

Loss

• Begin: 2.50
• End: 2.48

Completed Data

• $\approx$ 207.08GB

Average Grad Norm

• 0.76

Progress

• 22.24%

Comment

At 9:10 this morning, we encountered the OOM issue once again. It's weird! We just resumed the training at 16:08. As this problem occurs frequently these days, we will try to find out the cause in the coming days🤔. If you have any ideas, please post them in the community!