In the en_core_web_trf model, using the entity_linker, the training loss does not decrease

[wh5938316]

I have completed the training of the first EntityLinker model following the instructions in tutorials/nel_emerson.

I noticed that the tutorials/nel_emerson uses the en_core_web_lg model, whereas the en_core_web_trf model uses transformers instead of tok2vec. Therefore, after I replaced the model with en_core_web_trf, I found that doc.vector is an empty array, which is as expected. So, I got the vector using doc._.trf_data.model_output.last_hidden_state[0, 0, :]. The subsequent steps have not changed from the process in tutorials/nel_emerson.

However, during the training of the entity linking model, the training loss did not decrease as quickly as before.

I know there must be something wrong with this approach, but I don't know the specific reason. I am a newbie in NLP and just getting into this field.

If I want to use transformers and the entity description of KnowledgeBase to convert vectors, can I use tok2vec?

Four years after the tutorials/nel_emerson was proposed in 2019, have there been any new developments in the field of entity linking?

Can anyone answer these questions for me, thank you very much!

[rmitsch]

Hi @wh5938316!

However, during the training of the entity linking model, the training loss did not decrease as quickly as before.

I know there must be something wrong with this approach, but I don't know the specific reason. I am a newbie in NLP and just getting into this field.

If I want to use transformers and the entity description of KnowledgeBase to convert vectors, can I use tok2vec?

My first guess here: the transformer wasn't trained with the objective of producing embeddings that reflect document similarity via cosine distance. tok2vec - which is not a transformer model - was trained with this objective, but not all embedding/language models necessarily are. Since cosine distance/similarity is used in spaCy's entity linking to determine document similarity, things may go awry here.

If you want to use transformers for this, I'd recommend using sentence-transformers by installing this spaCy extension.

Four years after the tutorials/nel_emerson was proposed in 2019, have there been any new developments in the field of entity linking?

Four years are a long time in this field, so it's difficult to summarize the developments here. In a nutshell - entity linking has moved towards leveraging embeddings even more since 2019, and there are more end-to-end approaches (i. e. without selecting a shortlist of viable candidates as a prior step, and instead directly choosing the entity to link from the entire set of available entities). An example for how this may look like with a generic set of underlying embeddings is DEER, which we are potentially integrating in spaCy v4 (which will feature other improvements w.r.t. entity linking too, such as support for customizable and external knowledge bases).

[wh5938316]

Thank you very much for your answer @rmitsch!

I noticed the BLINK model released by Facebook in 2019. The idea of BLINK is also to quickly generate candidates through an alias table and use neural networks to select the most likely ones. Although BLINK has an innovative bidirectional architecture, it seems to be fundamentally the same as spacy's entity_linker. Is this the case?

In addition, you mentioned the DEER architecture. As a novice in NLP, I am not capable of building a new model from scratch through papers. I am very much looking forward to your mention of spacy 4.0 and the new entity_linker using the DEER architecture. Is there a timeline for these plans? Or are there any open source projects that you know of that have implemented the DEER architecture?

[rmitsch]

Although BLINK has an innovative bidirectional architecture, it seems to be fundamentally the same as spacy's entity_linker. Is this the case?

On a basic level they both embed candidate descriptions and mention contexts and then select entities to link based on similarities in a dense embedding space. The mechanisms used to do so vary however, BLINK is certainly more sophisticated than the spaCy implementation (e. g. cross-attention between entity description and mention context, using a vector store for fast retrieval etc.).

Is there a timeline for these plans?

Not one we're confident in announcing yet, no.

Or are there any open source projects that you know of that have implemented the DEER architecture?

As far as I know, no. Especially not production-ready stuff.