Sentence similarity detection

[hyperknot]

Hi, I'd like to compare a grammar corrected text's before - after state. As a first step I'm splitting the text to sentences and then I'd like to match the sentences to pairs.

Is there any way to do sentence similarity detection when using xx_sent_ud_sm? Or I'd need to use en_core_web_lg, and thus limit this use-case to single language (or a few languages which have lg model).

I mean I think the sentence similarity detection should work even with a non-NLP approach, just by splitting words by spaces and comparing those. Does Spacy support something similar?

[rmitsch]

Hi @hyperknot! By default spaCy uses word embeddings for sentence similarity detection. However it's easy to customized this behavior to calculate similarity according to your own preferences. Check out the docs on user hooks for similarity calculation :)

[hyperknot]

Thanks for the response. I've seen in Universe that there exists two packages:
https://github.com/MartinoMensio/spacy-universal-sentence-encoder
and
https://github.com/MartinoMensio/spacy-sentence-bert

Which allow me to use USE and SBERT with spaCy.

If I understand correctly, when using those packages I just load the modell and run nlp() on each sentence, which gives me a vector?

[rmitsch]

Yes, see the usage docs (note that both are third-party extensions and not maintained by the spaCy team).