How to create an Entity Linker without Knowledgebase

[ecemture]

I have a custom dataset, and I trained a NER transformer model with it. I want to implement an Entity Linker. However, spacy’s entity linker approach requires a knowledge base.

I couldn’t create a knowledge base because I didn’t have any aliases or prior probabilities in my dataset. To overcome this issue, I started to use a MongoDB API and implemented a get_candidates() function. This get_candidates function returns entity_id, entity_vector, and span. Here is the output format of it:

[
{
"_id": "string",
"wiki_id": "string",
"prior_prob": 0,
"frequency": 0,
"embeddings": "Unknown Type: list"
}
]

I am hoping to replace Spacy’s get_candidates function, so whenever the entity_linker sends a request to the knowledgebase, it will get a response from my database. I want to train the entity_linker model by using this structure. The problem is get_candidates() function of spacy returns a Candidate object which takes the knowledge base as input.

So I wonder, is there any way to use this function without creating a knowledge base? How many functions should I rewrite If I want to create a Candidate() object with Knowledgebase equal to None?

[rmitsch]

Hi @ecemture, to make sure I understand correctly: are you looking to write your own entity linking component or do you want to use spaCy's implementation?

[ecemture]

Hi @rmitsch I want to use spaCy's implementation. I think I need to do some changes to use it but I am not looking to write my own entity linking component.

[rmitsch]

In that case I'm afraid you won't be able to do this without constructing a KnowledgeBase object, as spaCy's entity linker relies on the KnowledgeBase reference being set in the Candidate objects it receives.

spaCy v4 will be more flexible in this aspect and allow for easier customization of/derivation from KnowledgeBase, so there the canonical solution to this would be to write a KnowledgeBase wrapper interacting with your external knowledge store (your MongoDB). If you are interested in how to do this, you can have a look at the dev branch in our wikid project, in which we implement a KnowledgeBase class for data from Wikipedia and Wikidata in a SQLite database.
Note: This isn't stable yet.