Find similarty in GERMAN sentences

[SimonF89]

Hi,
iam new to spacy and trying to find similar sentences in my database. We store requirements in a sql-database and want to find potential redundances. Therefore we would like to use something like BERT or spacy to find similarities also in the meaning, using different words.

We tried the standard-spacy similarity function but got strange results:

import spacy  # 2.1.8
nlp = spacy.load("de_core_news_md")

sentenceA = nlp("Das System muss 10 Bierkisten transportieren können.")
sentenceB = nlp("Das Auto sollte 20 Km/h schnell fahren.")
sentenceC = nlp("Das System muss so gestaltet sein, dass es bis zu 100kg tragen kann.")
sentenceD = nlp("Peter soll bis morgen 10 Bierkästen transportieren.")
sentenceE = nlp("Falls der Peter morgen geht, muss das System dem Anwender die Möglichkeit bieten, 10 Bierkisten transportieren können.")

sentenceA .similarity(sentenceB) ==> 83%  --> not similar
sentenceA .similarity(sentenceC) ==> 86%  --> not similar
sentenceA .similarity(sentenceD) ==> 83%  --> similar
sentenceA .similarity(sentenceE) ==> 91%  --> similar

Is there a other way to get more precise results? As you can se AB, AC and AD have almoste the same results even though B and C arent similar to A.

Looking forward for your recommendations =)

[adrianeboyd]

Two issues here:

1. I noticed recently that the vectors distributed with the German models have a lot of missing vocabulary, so I wouldn't recommend using the de_core_news_md vectors.

   >>> for t in sentenceA:
   ...   print(t.text, t.has_vector)
   ... 
   Das True
   System True
   muss True
   10 False
   Bierkisten False
   transportieren True
   können True
   . False
   

   Alternate vector sources can have different tokenizations which can cause minor problems, but it's pretty easy to use spacy init-model to create a spacy model with vectors from word2vec text format (not binary format!) vectors. As an example using fasttext vectors (from https://fasttext.cc/docs/en/crawl-vectors.html / German: https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.de.300.vec.gz):

   spacy init-model de de_fasttext_vectors -v cc.de.300.vec -vn cc.de.300
   

   You can reduce the number of unique vectors with -V to save space. The _md models use -V 20000.

2. Spacy is just averaging over the token vectors for doc.similarity() which may not lead to a very useful comparison for your task. Antonyms often have similar word vectors, too, so you'll have to experiment and see what works well. I'd recommend looking into methods related to sentence embeddings and semantic textual similarity. If you don't mind working with research code, this is an example that looks promising and has a pretrained multilingual model that includes German: https://github.com/UKPLab/sentence-transformers

[SimonF89]

thx the ukplap solution helped a lot!!!