Extend NER Model with Custom Entity

[Btibert3]

I want to take the existing NER model that comes with en_core_web_md and simply extend it for a custom entity. I have trained a model with the with the CLI tooling for spacy 3.2, and while it performs well for my custom entity, I want to update/add this to the entities already trained in en_core_web_md.

Is this possible via the CLI training? I have seen some code samples that are not using the CLI, but they also appear to be for earlier versions of spacy ( < 3.0).

[Btibert3]

So in doing some digging, it looks like a recommendation would be to use the existing model to generate a dataset for the entities that I want to keep. From there, include this dataset with my new entity to re-learn an entirely new NER model from scratch?

#5134 (comment)

[polm]

Yes, we recommend training a fresh model rather than directly extending existing models.

Retraining a model with silver data can run into data quality or quantity issues, so another option is to train a separate NER component and put it in the same pipeline. When you have two NER models in one pipeline you can allow one to overwrite the other or not, and what configuration makes sense depends on the type of entities. See the double NER project for some discussion of this.

[Btibert3]

Interesting, I will give that a look, thanks!

[adrianeboyd]

And while we wouldn't really recommend it for adding new entity types (but it's fine for fine-tuning for existing entity types as long as you annotate all the entity types in your data), we do have a demo that extends an existing NER model:

https://github.com/explosion/projects/tree/v3/pipelines/ner_demo_update