Skip to content

01-semantic-similarity-primer.md

Latest commit

 

History

History
116 lines (86 loc) · 5.47 KB

01-semantic-similarity-primer.md

File metadata and controls

116 lines (86 loc) · 5.47 KB
Raw

1. A Semantic Similarity Primer

Estimated reading time: 12 minutes. Prerequisites: none.

1.1 What is semantic similarity?

Semantic similarity quantifies how close in meaning two linguistic items are. The items can be words ("car" vs. "automobile"), phrases, sentences, or whole documents. The output is usually a number — often normalized to [0, 1] — where a higher value means "more similar in meaning."

It is worth separating two ideas that are often confused:

  • Similarity — items are of the same kind and interchangeable in meaning: car / automobile, gem / jewel. This is what SimLex-999 deliberately measures.
  • Relatedness — items are associated but not interchangeable: car / road, coffee / cup. WordSim-353 mixes similarity and relatedness.

A measure tuned for relatedness will look "wrong" if you grade it on a similarity benchmark, and vice versa. Always know which one your dataset encodes.

1.2 Why it matters

Semantic similarity is a building block, not an end in itself. It appears in:

  • Information retrieval — ranking documents/passages by closeness to a query.
  • Question answering — selecting the answer sentence most similar to the question.
  • Paraphrase & duplicate detection — "are these two questions the same?"
  • Knowledge graphs / ontology matching — aligning entities and concepts.
  • Machine-translation evaluation — comparing a hypothesis to a reference.
  • Biomedical & clinical NLP — mapping mentions to standardized concepts.

Because it is a building block, small accuracy gains in the similarity component propagate to every downstream system that uses it.

1.3 The main families of measures

There is no single "semantic similarity algorithm." There are families, each with a different notion of meaning.

(a) Lexical / string measures

Operate on the surface form of the text: character n-gram overlap (Jaccard, Dice), edit distance (Levenshtein), longest common subsequence. No external knowledge required, so they are fast and language-agnostic, and they excel at near-duplicate / typo detection. But they are blind to meaning: car and automobile share almost no characters yet are near-synonyms.

In this toolkit: measures.LEXICAL_MEASURES (jaccard_2g, dice_2g, overlap_2g, levenshtein, lcs, length_ratio).

(b) Knowledge-based (ontology) measures

Use a structured resource — most famously WordNet — and measure distance in its taxonomy:

  • Path-based: path (inverse path length), Wu–Palmer wup (depth of the least common subsumer), Leacock–Chodorow lch.
  • Information-content (IC) based: Resnik res, Lin lin, Jiang–Conrath jcn, which weight concepts by how specific they are (estimated from a corpus).

These capture genuine taxonomic meaning (car / automobile are close) but are limited to the vocabulary and structure of the ontology.

In this toolkit: measures.wordnet_similarity(a, b, method=...).

(c) Distributional / embedding measures

"You shall know a word by the company it keeps." Represent items as vectors learned from co-occurrence (word2vec, GloVe, fastText) and take the cosine of the vectors. They scale to huge vocabularies and capture soft, graded similarity, but they conflate similarity with relatedness and inherit corpus biases.

(d) Transformer / contextual measures

Contextual encoders (BERT, RoBERTa, Sentence-BERT) produce embeddings that depend on context, and are the current state of the art for sentence similarity (STS). They are powerful but heavier (model downloads, more compute) and harder to interpret.

These families are documented here and easy to plug in (any f(a, b) -> float), but the runnable defaults in this repo stay lightweight so every example reproduces on a laptop.

1.4 How is a measure evaluated?

The standard protocol for word/sentence similarity is:

  1. Take a benchmark of item pairs, each with a human gold score (the mean of several annotators' ratings).
  2. Run your measure on every pair to get predicted scores.
  3. Compute the correlation between predicted and human scores.

Because humans agree on ranking more than on absolute values, the field reports Spearman (and often Pearson/Kendall) correlation rather than raw error. See Evaluation Metrics.

from similarity_ensemble import measures, metrics, datasets
df = datasets.load("mc30")
pred = [measures.jaccard_char_ngrams(a, b) for a, b in zip(df.word1, df.word2)]
print("Spearman:", round(metrics.spearman(pred, df.gold), 3))

1.5 The punchline that motivates this repo

Each family above is "right" about a different aspect of meaning, and each is "wrong" in different places. That observation — that the errors are diverse — is exactly what makes ensembles worthwhile. That is the subject of the next chapter.

References

  • Rubenstein, H., & Goodenough, J. B. (1965). Contextual correlates of synonymy. CACM, 8(10).
  • Miller, G. A., & Charles, W. G. (1991). Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1).
  • Resnik, P. (1995). Using information content to evaluate semantic similarity in a taxonomy. IJCAI.
  • Hill, F., Reichart, R., & Korhonen, A. (2015). SimLex-999. Computational Linguistics, 41(4).
  • Martinez-Gil, J. (2022). A comprehensive review of stacking methods for semantic similarity measurement. Machine Learning with Applications, 10:100423.