Resuming model training from last checkpoint

[crtnx]

I am aware of the discussion in this post #8176 however I'd like to clarify some points. When I do resume the training from the last checkpoint, the epoch count starts over and potentially the resumed training losing a part of a progress done before? Here the output when the training just started:

E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0         851.85   1141.28    1.58    0.99    3.86    0.02
  0     200      379219.13  145062.73   67.59   71.19   64.34    0.68
  0     400      269841.19  96900.35   72.41   72.63   72.20    0.72
  0     600       88137.48  66657.66   93.87   95.24   92.55    0.94
  0     800       44777.95  52530.02   97.87   97.91   97.83    0.98
  0    1000       42356.65  49182.96   98.33   98.47   98.18    0.98
  0    1200        2781.13  35941.77   97.72   97.91   97.53    0.98
  0    1400        3161.14  34798.89   99.34   99.36   99.32    0.99
  0    1600        1032.14  34074.89   99.45   99.41   99.49    0.99
  0    1800         625.08  32244.73   99.65   99.64   99.66    1.00

Here is the resumed training:

=========================== Initializing pipeline ===========================
[2021-12-12 10:04:56,075] [INFO] Set up nlp object from config
[2021-12-12 10:04:56,084] [DEBUG] Loading corpus from path: /content/drive/MyDrive/Colab Notebooks/corpus/dev.spacy
[2021-12-12 10:04:56,085] [DEBUG] Loading corpus from path: /content/drive/MyDrive/Colab Notebooks/corpus/train.spacy
[2021-12-12 10:04:56,086] [INFO] Pipeline: ['transformer', 'ner']
[2021-12-12 10:04:56,086] [INFO] Resuming training for: ['ner', 'transformer']
[2021-12-12 10:04:56,091] [INFO] Created vocabulary
[2021-12-12 10:04:56,092] [INFO] Finished initializing nlp object
[2021-12-12 10:04:56,092] [INFO] Initialized pipeline components: []

E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0           0.83    122.99   99.65   99.64   99.66    1.00
  0     200         553.36  31544.02   99.67   99.68   99.66    1.00
  0     400         617.94  30809.99   99.50   99.52   99.49    1.00
  0     600        1545.63  29539.56   99.63   99.66   99.59    1.00
  0     800         943.73  26552.97   99.50   99.52   99.48    0.99
  0    1000        1088.28  23256.71   99.70   99.71   99.68    1.00
  0    1200         902.59  19516.21   99.72   99.73   99.71    1.00
  0    1400         427.07  15410.02   99.70   99.70   99.69    1.00

Another point which is not clear - should I specify 'model-last' or 'model-best' as a source for resume?

[ljvmiranda921]

Hi @crtnx ,

When I do resume the training from the last checkpoint, the epoch count starts over and potentially the resumed training losing a part of a progress done before?

Resuming training shouldn't lose the progress but yes, the epoch count in the UI starts over. So if your model-last stopped at epoch 5, resuming it is akin to doing epoch 6, although the UI will show epoch 0. Just note that when you resume training, the optimizer is reset. This will change the final results slightly compared to just running it in one training session.

Another point which is not clear - should I specify 'model-last' or 'model-best' as a source for resume?

Personally I use model-last when I still want to keep optimizing parameters on a wider search space (or simply if I just want to pause and continue for some reason). Then I resume from model-best if I want to keep optimizing on the best found parameters. It's always up to you and your system to decide.

Also, I'm taking a look at your results and it seems unnecessary to resume training further at 99.7 (it's pretty high already and you might want to check overfitting instead) 🤔? Unless you're running this on a toy dataset. 😉