Entity Linker with Transformer Listener

[gblanco10]

I have a pretrained pipeline composed by a transformer and NER components and I am trying to create an Entity Linker able to use embedding representation produced by the transformer rather than using a CNN.
I am adding the Linker to the pipeline as

entity_linker = nlp.add_pipe("entity_linker",config = {
"labels_discard": [],
   "n_sents": 0,
   "incl_prior": True,
   "incl_context": True,
   "model": {
       "@architectures": "spacy.EntityLinker.v2",
      "tok2vec": {
               "@architectures" : "spacy-transformers.TransformerListener.v1",
               "pooling" : {"@layers":"reduce_mean.v1"},
               "upstream" : "*"
            },
         "nO": 768
      },
   "entity_vector_length": 768,
   "get_candidates": {'@misc': 'spacy.CandidateGenerator.v1'},
   "threshold": None,
})

but I receive dimension unset Error

  File "/home/blanco/wiki_kb/.venv/lib/python3.9/site-packages/spacy_transformers/layers/listener.py", line 64, in forward
    width = model.get_dim("nO")
  File "/home/blanco/wiki_kb/.venv/lib/python3.9/site-packages/thinc/model.py", line 175, in get_dim
    raise ValueError(err)
ValueError: Cannot get dimension 'nO' for model 'transformer-listener': value unset

Looking online I found that this last problem typically occurs when you try to finetune a late component in the pipeline while freezing the transformer but in my case I am no quite close to that point.

My final aim would be to have a Linker that uses transformer embedding for entity and context and (maybe learning a linear layer for reprojection to a lowest dimension or in case no train at all) compares embedding coming from a knowledge base whose embedding representations have been created using the same pretrained pipeline.

Thanks in advance to everybody who will reply and help me

Current Environment

• Python 3.9.13
• spacy 3.4.4
• spacy-alignments 0.9.0
• spacy-legacy 3.0.12
• spacy-loggers 1.0.4
• spacy-transformers 1.2.1

[adrianeboyd]

It's hard to know for sure without seeing your full config, but if you want to use a frozen transformer as input to another component, be sure that you're using spacy v3.5.0+ and spacy-transformers v1.2.2+ due to some bug fixes and be sure that you've added it to [training.annotating_components].

If that doesn't help, can you provide the full config and training steps that you're using?

[rmitsch]

Thanks for reporting back. We will take a look at this.

[gblanco10]

Hi @rmitsch, have you had a chance to look at the experiment results? Thanks

[rmitsch]

Sorry for my late response! No, not yet. It's on my to-do list, but I haven't gotten around to it yet. I'll update here :)

[rmitsch]

Short update - I haven't gotten around to this yet, but I should be able to do so soon. Will keep you in the loop!

[rmitsch]

My apologies for the late response! So, I finally ran the NEL benchmark with the current main of https://github.com/explosion/projects. I ran every single step without modifications and ended up with reasonable, non-zero scores. Are you sure you ran the vanilla configuration on a fresh venv?

[jkgenser]

We are hence currently working on modifying our NEL architecture so that both KB and text input embeddings are fine-tuned to this end.

@rmitsch This sounds very interesting. I've had success with current NEL model and interested in whether you have documented intended plans here or literature or other resources the new work is inspired by?

[rmitsch]

Happy to hear that the current NEL model worked for you! We have no publicly available documentation on the intended plans. What I can tell you at this point is that we are thinking of experimenting with the DEER architecture as introduced here.

[jkgenser]

Will look into that. For my use case, I've used a fuzzy search candidate generator because there is a large space of potential aliases and most are not present in training data. i.e. "a. fib, "a. fib.", etc. which map to "atrial fibrilation" This approach has worked well but I've also been debating using ANN as part of candidate generation as well as debating removing the linker component entirely and fine-tuning a transformer pipeline on NER, then taking the finetuned transformer and then encoding mention+context as well as kb entity + description and doing ANN as the entity resolution step itself; kind of like sciscpacy did but instead of using tfidf 3/4 chargrams using the transformer encoding. I suppose CNN encodings would work similarly.

One of the biggest issues I find is that that I have the encode the entity descriptions into the knowledge base before fine-tuning on NER+NEL task which is updating weights back into my transformer (or CNN I suppose) and therefore learning a better encoding of mentions + descriptions.

Do you think the approach I describe is reasonable stopgap before the DEER approach is implemented into spacy? (FYI: @svlandeg also since I know you're another spacy expert on KB things - no pressure to weigh , tagging you in case hearing from another user and use case is useful)

That paper looks interesting because it would involve learning representations for an NN search which also seems to be the direction a lot of the literature is going. Looking forward to it!

[rmitsch]

One of the biggest issues I find is that that I have the encode the entity descriptions into the knowledge base before fine-tuning on NER+NEL task which is updating weights back into my transformer (or CNN I suppose) and therefore learning a better encoding of mentions + descriptions.

IMO one of the advantages of DEER is that the "fine-tuning" happens via a separate model on top of the models you use to infer your base embeddings. I. e. it's not really fine-tuning in that sense, but mapping the embeddings for entity descriptions and textual mentions to another embedding space, with the objective of maximizing the similarity of the resulting embeddings of the description of entity E and the mention M if M should resolve to E. So you have a relatively light-weight model to adjust the embeddings you're getting out of your transformer model, and it's learning to do so for entities and mentions in tandem. At inference time an ANN vector search operation is sufficient to retrieve the most relevant candidates (it might be beneficial to add another linker model on top, we'll test that).

Do you think the approach I describe is reasonable stopgap before the DEER approach is implemented into spacy?

It sounds reasonable to me. If you do that, we'd be happy to hear about your results!