Training Log 2022-06-28

[zh-zheng]

CPM-Live Training Log (June, 28)

Time: June, 28 2022 16:00

Recorder: @zh-zheng

Loss

• Begin: 2.62
• End: 2.65

Completed Data

• $\approx$ 348.90GB

Average Grad Norm

• 0.90

Progress

• 37.46%

Comment

Another peaceful day😁. Although there are still some fluctuations in the loss value, its downward trend remains unchanged. Today I notice an interesting paper, in which researchers use OpenAI Codex to solve university-level mathematics problems. What's more, the explanation of the solution code can also be automatically generated. As you can see, when a model performs well and is explainable, it looks charming! We will also make this our goal when training CPM-Live.💪

[XiaoqingNLP]

hi，想问一下，规划书里面提到的"预计整个训练周期为5个月"，如果损失处于这种波动状态，咱们实际计划训练到什么条件下停止训练？
训练时间对大模型来说，是成本和收益的trade-off；
按网站上显示的进度还需要训练60%，还需要2个月时间

[zh-zheng]

拉长时间来看，我们的训练损失目前是在缓慢下降的。按照我们的计划，第一轮训练会把200B tokens过完一遍。