Using spacy's NER to for Entity Recognition of custom entities #9940
-
|
Hi!So I am currently working on building a custom entity recognizer using spacy. Whilst preparing the training examples, For tagging the data using the BILOU scheme, should I only add the new custom entity tags in my data while keeping the tags for the other tokens as 'O', or should I tag those as well? I only want to add new entities to the existing NER. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 6 replies
-
|
It sounds like the best thing is to label just your entities and train a separate NER model to combine with the existing one. See the double NER example for notes on how that works. Also note that while you say you want to add the new entity to the existing NER model, I would recommend against that. If you only add your new entities as training data you'll run into the "catastrophic forgetting" problem, so you'd have to add data for all entities you want to keep using. But that would be a lot of work, can be hard to get right, and having all the entities in one model isn't necessarily better than having separate models. |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure what you mean by "the first route". My suggestion is you label just your new entities and use that to train a model. Then you can combine that model with the existing one as a separate component. The example project does something similar. |
Beta Was this translation helpful? Give feedback.
-
|
Got it. I'm sure this will solve my problem.Thanks a lot! |
Beta Was this translation helpful? Give feedback.
-
|
Apart from the size or the size/RAM considerations mentioned in the link above, are there any other aspects that need to be considered with the above approach? |
Beta Was this translation helpful? Give feedback.
-
|
I think the RAM considerations are pretty important. Besides that, if you have a lot of models, I'd be suspicious of whether they were actually independent. |
Beta Was this translation helpful? Give feedback.
-
|
yeah, the labels are mutually exclusive. and if there are related labels, I add it to the existing NER model and rehearse the model (pseudo-rehearsal). |
Beta Was this translation helpful? Give feedback.
It sounds like the best thing is to label just your entities and train a separate NER model to combine with the existing one. See the double NER example for notes on how that works.
Also note that while you say you want to add the new entity to the existing NER model, I would recommend against that. If you only add your new entities as training data you'll run into the "catastrophic forgetting" problem, so you'd have to add data for all entities you want to keep using. But that would be a lot of work, can be hard to get right, and having all the entities in one model isn't necessarily better than having separate models.