Upgrading from static vectors to contextual vectors

[Gitclop]

Hey, i have build a semantic-similarity pipeline using the following steps:
Load a set of n documents. For each document do the following:

• Tokenisation -> Bag-of-Words
• Generate document-vector from the bag-of-words
• Build an annoy (aproximate nearest neighbour) index and find the x closest documents using euclidian distance

the static vectors have been trained with word2vec, and i want to compare the accuracy of my similarity task between those static vectors and different transformer models.

For the static vectors, i use a trained spacy-model and this code:
data.loc[:, ('Vektoren')] = data['bag_of_words'].map(lambda s: nlp(s).vector)

changing my vectors within the pipeline is probably not as easy as:

nlp = spacy.load('de_dep_news_trf')
data.loc[:, ('Vektoren')] = data['bag_of_words'].map(lambda s: nlp(s)._.trf_data.tensors)

so how would i upgrade to transoformer models? Is it usefull to still extract the document-vector or are there better ways of finding the x nearest neighbours/documents?

[rmitsch]

Hi @Gitclop, if you just want to extract transformer embeddings and copy them into your dataframe, you are pretty close already:

nlp = spacy.load('de_dep_news_trf')
data.loc[:, ('Vektoren')] = data['bag_of_words'].map(lambda s: nlp(s)._.trf_data.tensors[1])

Is it usefull to still extract the document-vector or are there better ways of finding the x nearest neighbours/documents?

As long as the underlying model has been trained to reflect document similarity in its embeddings, yes. Note that the pretrained transformer pipelines offered by spaCy aren't. For the purpose of comparing documents by their embeddings we recommend sentence-transformers.

[Gitclop]

Thanks a lot!
I tried loading the spacy-sentence models from https://github.com/MartinoMensio/spacy-sentence-bert but every model i tried downgrades my spacy installation to 3.0.9 are there any up-to-date models available for spacy?

[rmitsch]

We don't maintain this project, I suggest asking there. You can of course always infer the sentence embeddings for your texts without spaCy and add them to your dataframe, if that's all you need.