"The model you're using has no word vectors loaded"- Even though all my tokens do have a vector representation

[e412]

Hello :)
I am trying to calculate the textual similarity between two dataframe columns in German: get the similaritiy between every row-item with every other row-item and return a Top10-Matching-List. As only 63% of my tokens do have vectors using the [de_core_news_lg]-spacy package, I am trying to create my own vectors for these OOV compunds. I have done this using floret and I am able to import my new vectors by calling the folder with the nlp(my_vector_folder) command. I am checking the result printing out the nearest neighbours of some of my "specific" words (model.get_nearest_neighbors('pzb')), which look good. Also 100% of my tokens now return "True" for the has_vector-function.
BUT when trying to calculate the items by using the .similarity() function i get the warning:
UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. en_core_web_sm, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.
similarities_per_item.append([i, z, (docs_processed_hergang[i].similarity(docs_processed_codetext[z]))])

Also the result performs realy bad. I get the same output as i am creating the top10 matches randomly. The result using the [de_core_news_lg]-spacy library performs much better even though all the important words are missing in there.

Do you have any idea what might cause this problem?

Thank you so much in advance.

Kind regards
Eva

[adrianeboyd]

If you're using spacy v3.2, you can get an incorrect W007 warning even if you have floret vectors, but if you're using v3.3+, this may mean that your vectors weren't initialized/loaded correctly. I hope you meant spacy.load(my_vector_folder) and not nlp(my_vector_folder)?

Be aware of the notes here about floret vectors: https://spacy.io/usage/v3-3#upgrading

It's hard to know what's going for sure without a concrete example including the model vectors. Just to make sure something didn't go wrong on import (although I think you should get an error with spacy init vectors --mode floret if the format was incorrect), double-check that nlp.vocab.vectors.mode is 'floret'?

I tried out some simple cases with fi_core_news_md and didn't notice any unexpected behavior around doc.similarity directly:

import spacy
nlp = spacy.load("fi_core_news_md")
nlp("paikka").similarity(nlp("paikkaa")) # 0.5902148855603732