Why labels from training data are added in "ner pipe" during incremental training?

[umayerr]

Currently I am trying to implement incremental training for ner pipe in spaCy. I am using en_core_web_sm model and trying to update the model by adding new dataset. I see in the code, all the labels from training data are added in pre-existing model’s ner pipe even if those labels are already in that pipe [that piece of code is given below]. I did not understand why it is necessary. My specific questions are:

1. What will happen if I skip this step? I am introducing new training data with existing labels from en_core_web_sm model only.
2. The code mentioned below adds same label multiple times. E.g. if label 'PERSON' comes 100 times in training data, it executes ner.add_label('PERSON') 100 times? Is it necessary? For the larger dataset it might increase the runtime.

for _, annotations in TRAIN_DATA:
    for ent in annotations.get("entities"):
        ner.add_label(ent[2])

[svlandeg]

Hi @umayerreza, where are you pulling this code snippet from? It is indeed unnecessary to call add_label with the same label over and over again. The function will check if the label is already known and won't do anything if it is, but you're right that it might still introduce a slight overhead. That said the overhead should only occur at the start of the training phase, and the actual training would be much longer in comparison. And you shouldn't have to do this at all when you're using the model for inference.

If you know all your labels up front, you can simply call add_label for each of them once instead of looping through the training data.

[umayerr]

Hi @svlandeg, thank you for you reply. Yes, I guessed something similar. I got this code here. The same is also suggested here too. I think this two references are based on spaCy v2. However, I updated the nlp.update() function in my code according to v3.