Is It Possible to Resume Training Via CLI in Spacy v3 (transformers)?

[user06039]

I was training a ner model with a config file and train and validation set , I trained for 10 epochs and model-best and model-last got saved and I use those model for inference.

python -m spacy train config.cfg --paths.train ./train.spacy --paths.dev ./valid.spacy --output models --gpu-id 0

Is there any way I can resume training from 10th epoch and continue on again.

[svlandeg]

This is possible, but you'll have to change your config slightly. Basically for the next round of training, you don't want to create new components/models, but you want to start from the already trained pipeline. In terms of config values, that means you don't want to use any "factories" anymore, but you'll want to "source" your components instead, see https://spacy.io/usage/processing-pipelines#sourced-components

So let's say you were training an ner component and that's saved in models/model-last, then you should be able to do something like this in your config:

[components.ner]
source = "models/model-last"
component = "ner"

You won't need to specify the model of this ner component anymore, because its settings will be loaded from the configuration file that is present in models/model-last.

If you have multiple components that you want to continue training, you can just "source" all of them.

[miguelwon]

This is not working for me. But I'm not sure if is related with my data. I trained a model with a particular dataset. Now, I want to resume training with new data, but this new data has adds also a new category. Is it possible?

I used the config.cfg file from the first training step and just replaced the "factory" line to "source", as you suggest. Unfortunately, I think it didn't work. The resulted ner is giving very wrong predictions, with f1 below 10.

[polm]

The advice here is for resuming training with the same configuration, including data.

Continuing training with new labels is supposed to work but can run into issues with catastrophic forgetting, so we usually recommend against it, see the FAQ. If you want to update anyway, see this example project.

Besides issues with forgetting, you might have an issue with your labels getting picked up.

Since your question is different from the main question here, if you'd like to discuss this more, it'd be better if you opened a new thread and linked back to this one.

[miguelwon]

Ok, is working after generating the config file with the script from the example problem. So far, it seems to be working fine.
(just one note that does not need a new thread: the only odd thing is that for the first few epochs the f1 for the new class was zero and then, in just two epochs, it jumped to a nice value of about 90)

[trivediashutosh]

I am also trying to retrain the spacy model on more data. I have the previously trained model-last file. Can you share the differene between config file in following two case: 1. When you trained spacy model for the first time and 2. When you used the last chepoint to train the spacy model again on new data.

[polm]

@trivediashutosh As covered in Sofie's comment, when you use the last checkpoint you change your config to source all the components.