Detect different references of same entities as one

[shrinidhin]

Hello!I have built a custom NER using spacy to detect custom entities. It detects all the entities as required. There are certain scenarios of different ways the same entity is referenced.

Scenario 1: Company name and its abbreviations Eg: Mahindra and Mahindra also referenced as M&M
Scenario 2: For multi word entities, Complete name of company is referenced once in the doc and thereafter it is referenced using the first word.
Eg: Tata Consultancy Services reported losses this quarter. Tata released a statement detailing its performance over the last quarter.
Here Tata Consultancy Services and Tata are correctly picked out as entities but these are the same entity i.e. both refer to Tata Consultancy Services.
Scenario 3: Company having two different names i.e. Company ABC and XYZ are talking about the same company or entity.

While I understand that the NER component as a whole only recognizes the entities, whether they are same or not is something that has to be handled post processing, I was thinking whether using an Entity ruler will be helpful here?What other approach can I use apart from maintaining a local dictionary with all possible references and checking via code?While the latter seems easier to implement, it seems time consuming to run everytime you pass a doc to detect entities.
Thank you!

[thomashacker]

Hello,
I think EntityLinking would be helpful for your usecase (https://spacy.io/api/entitylinker) it connects found entities to unique identifiers.

[shrinidhin]

Thank you!Can you help me understand Why are we storing an nlp object to the disk?In my case can I use my trained NER model object as input for the nlp=spacy.load() part?

[polm]

I'm not really sure what you're referring to, could you link to the code? You should be able to use your own NER model just fine.

[shrinidhin]

In the create_kb.py file, We are storing the nlp object along with our kb to the disk. Since I have an already trained NER model, already stored locally, i wanted to clarify I can use the same.
Thank you very much for your help!

[polm]

Ah, yes, you can ignore the steps related to saving the nlp object there if you already have a pipeline, since that's just for adding the entity ruler and sentencizer.

[shrinidhin]

Great!Thank you!