Trouble Creating Vectors from Recognized Entities

[mac389]

I trained a model on curated text, but when I cannot retrieve vectors for entities even though those entities are innlp's vocabulary.

I want to get those vectors to train a custom Named Entity linker.

Example:

 sample_string = "DNP is a thing."
 for ent in doc.ents:
     print(f"Named Entity '{ent.text}' with label '{ent.label_}'") 
     #prints Named Entity 'DNP' with label 'SUBSTANCE'
 print(nlp("DNP").vector) -> # []
 print(nlp.vocab.strings["DNP"]) # -> 12623617258395969273
 print(nlp.vocab.vectors[nlp.vocab.strings["DNP"]]) # -> Could not retrieve vector for key 12623617258395969273.'
 print("DNP" in list(nlp.vocab.strings)) #-> True

[polm]

It's normal to have words in vocab.strings that don't have a vector in vocab.vectors. vocab.vectors is for word vectors, which spaCy does not train; there's a fixed set loaded with a pipeline. The small and TRF pipelines we distribute do not have word vectors.

The .vector for a Doc (which is different from the vector for a token) would be empty like in your results if you're using a TRF model. The tensors from Transformers are in doc._.trf_data.

The NER component does not produce vectors, those would be created by a tok2vec or Transformer component.

If you are training using the spaCy Entity Linker component you don't need to manually get the vectors, they should be read in automatically from up the pipeline.

[mac389]

It's normal to have words in vocab.strings that don't have a vector in vocab.vectors. vocab.vectors is for word vectors, which spaCy does not train; there's a fixed set loaded with a pipeline. The small and TRF pipelines we distribute do not have word vectors.

Thanks, this is helpful. I assume this may also hold for smaller custom models.

If you are training using the spaCy Entity Linker component you don't need to manually get the vectors, they should be read in automatically from up the pipeline.

You mean using code like?

from spacy.pipeline import EntityLinker
entity_linker = EntityLinker(nlp.vocab, model)

But this doesn't address how add_entry requires a vector, if I understand correctly.

The example for custom entity linking accesses .vector, when loading nlp from en_core_web_lg to add entries to the KnowledgeBase (Line 52 in el_tutorial.py).

What should my approach be if the words in the descriptions for a KnowledgeBase entry don't have an associated vocab.vector? What vectors should I use?

[polm]

Thanks, this is helpful. I assume this may also hold for smaller custom models.

It's not about whether a model is small or not, it's just about the presence or absence of word vectors.

You mean using code like?

Sorry, I wasn't addressing the case of building a KnowledgeBase here. What I was referring to was the tok2vec structure of the pipeline.

The easiest thing for you to do is to just use the large model like the tutorial project does. If you want to use Transformers we usually recommend the sentence transformers library.

[mac389]

Sorry, I wasn't addressing the case of building a KnowledgeBase here. What I was referring to was the tok2vec structure of the pipeline.

Let me try to rearticulate my issue, because I think my first post was not clear.

I have a knowledge base structured exactly as in Sophia's Emerson example. According to that example, I need to pass nlp(ENTITY_DESCRTIPTION).vector to kb.add_entity. But, nlp(ENTITY_DESCRIPTION).vector fails because not all vocab.strings have vocab.vectors. I need to overcome this error.

The easiest thing for you to do is to just use the large model like the tutorial project does.

I trained a custom ner starting with spacy.blank('en') because a model using en_core_web_lg does not recognize the entities I need it to, which are technical biochemical names. I validated that my custom nlp recognizes the entities I want it to.

1. Should I create a model that starts with en_core_web_lg? Why would that model have vector representations for out-of-vocabulary technical words? It doesn't seem that adding my custom model would help. We've already established it doesn't add vectors.
2. Should I pass a different vector to the entity_vector field of kb.add_entity? For example create them with gensmin or the BERT encoder you linked to?
3. Is this issue with extending Sophie's NEL example on technical data sets not mentioned because it's a very specific use case, or am I running into an issue because I'm missing something fundamental about SpaCy?

[polm]

Ah, it does seem like I got my wires a little crossed here.

So the issue is not only that the model you are using gives an empty list instead of vectors, but also that even if you did have a model with vectors, your terms would be OOV and thus get empty vectors.

First I'd like to step back and ask, are you sure you need NEL? NEL will differentiate between multiple different terms a single entity could refer to. I would expect technical biochemical names to be unambiguous by default - maybe they aren't, but I want to check.

Next, to address why this didn't come up - I don't think it's a very specific use case, but I would expect most entities that could refer to multiple things would not be entirely OOV terms. It's also worth noting the NEL tutorial is an introduction, and not designed to cover every case. (It might be helpful for my understanding if you gave some examples of the kinds of problems you want NEL to solve for you.)

So, how can you get vectors?

If you use a model with word vectors, be it MD or LG, you will get vectors for all words or spans. However for OOV terms the vectors will be all zeros, which will not be useful if you have a lot of them. (The SM model will also give vectors for spans due to the fallback behavior of .vector, but these vectors won't be great.)

The simplest thing you can do for reasonable results is train your own word vectors using Gensim or similar. That should give you better coverage of your specific terms. What I'd recommend is using floret, described here, to get subword coverage, which will be important if you have a lot of sparse terms.

You can also use the spacy-sentence-embeddings project I linked to; that should work fine, though I haven't heard of anyone using it with NEL yet.