Estimated reading time: 12 minutes. Prerequisites: Chapter 1.
You cannot improve what you cannot measure honestly. This chapter explains the metrics this toolkit reports, when to use each, and how to compare methods without fooling yourself.
Given predicted scores ŷ and human gold scores y over n pairs:
Measures the linear relationship between ŷ and y.
r = cov(ŷ, y) / (σ_ŷ σ_y), in [-1, 1].
Sensitive to the exact values and to outliers; assumes a linear relation.
Pearson correlation computed on the ranks of ŷ and y. It asks: does the
method order the pairs the way humans do? It is invariant to any monotonic
rescaling of the scores — which is why it is the headline metric for
similarity/STS tasks (you rarely care about the absolute number, only the order).
A rank correlation based on the proportion of concordant minus discordant pairs (τ-b corrects for ties). More robust and more interpretable as a probability of correct ordering, but more conservative (smaller magnitudes than ρ).
from similarity_ensemble import metrics
metrics.pearson(pred, gold)
metrics.spearman(pred, gold) # report this by default for similarity
metrics.kendall(pred, gold)Rule of thumb: report Spearman as primary, with Pearson and Kendall alongside. If Pearson ≫ Spearman, your scores are linearly aligned but mis-rank pairs; if Spearman ≫ Pearson, the relation is monotonic but non-linear (a calibration/scaling step may help).
When the absolute value matters (e.g., you feed the score into a threshold), report error too. Because measures live on different scales, the toolkit min-max normalizes both vectors before computing MSE/RMSE so the numbers are comparable.
metrics.rmse(pred, gold) # lower is betterA correlation computed on 30 pairs (MC30!) is a noisy estimate. Reporting
ρ = 0.81 with no uncertainty hides how fragile that is.
Resample the pairs with replacement many times, recompute the metric, and take percentiles. This needs no distributional assumptions.
point, lo, hi = metrics.bootstrap_ci(pred, gold, metric=metrics.spearman)
print(f"Spearman = {point:.3f} (95% CI [{lo:.3f}, {hi:.3f}])")On small datasets these intervals are wide — which is exactly the honest message.
When 2000 resamples are too slow (or you want a sanity check), the Fisher
z-transform gives a closed-form interval. For Spearman's rho the standard error
on the z scale is about 1.03/sqrt(n-3):
lo, hi = metrics.fisher_ci(0.818, n=30) # -> roughly (0.64, 0.91)That interval, on MC-30, is wider than most published between-method deltas on that dataset. Keep it next to any small-n table you publish.
results = evaluation.benchmark(pairs, gold, with_ci=True) # CI columns built in"Method A scores 0.81, B scores 0.79, so A wins" is not a valid conclusion without a test — and because both methods are evaluated on the same pairs, their errors are correlated. The correct tool is a test for dependent correlations that share a variable (the gold): Williams's modification of the Steiger test.
# r_Ay = corr(A, gold); r_By = corr(B, gold); r_AB = corr(A, B)
p = metrics.steiger_williams_test(r_Ay, r_By, r_AB, n=len(gold))
print("Significant difference?" , p < 0.05)A nonparametric alternative that needs no correlation algebra is the paired bootstrap: resample the pairs jointly for A, B and the gold, recompute the delta each time, and see how often it crosses zero. Because both methods are always evaluated on the same resample, their shared noise cancels:
delta, p = metrics.paired_bootstrap_test(pred_a, pred_b, gold)evaluation.summarize() runs the Williams test automatically between the best
ensemble and the best single measure, so the printed delta always comes with a
p-value.
One more way to fool yourself, common enough to deserve its own heading: the "best single measure" row of a results table is usually the max over all base measures computed on the evaluation data. That max is an upwardly biased estimate of any measure's true skill (selection bias grows with the number of candidates and shrinks with dataset size), and it faces no cross-validation while the supervised ensembles do. The fair single-measure opponent is selection on training folds, scored out-of-fold:
preds = evaluation.cross_val_best_single(X, gold, k=5)benchmark() includes this as the best_single_cv row (kind baseline) by
default. Report both rows: the oracle for continuity with the literature, the
honest one for the actual comparison.
A supervised ensemble fit and then evaluated on the same pairs will look great and generalize poorly. The benchmark harness avoids this by reporting supervised ensembles on out-of-fold predictions (k-fold cross-validation):
from similarity_ensemble import evaluation
results = evaluation.benchmark(pairs, gold, cv_folds=5) # honest by constructionUnsupervised algebraic ensembles need no split; supervised ones (regression, genetic) are CV-scored automatically.
A serious benchmark of similarity systems should also consider, where relevant:
- Calibration — do predicted scores match observed similarity proportions?
- Runtime & memory — a tiny accuracy gain may not justify a 100× cost.
- Robustness — performance under noise, OOV words, domain shift.
- Diversity — for ensembles, how decorrelated the members are.
- Interpretability — can you explain a given score?
These are discussed in the FAQ and are natural axes for contributions.
- Report Spearman (primary) + Pearson + Kendall.
- Show a confidence interval, especially on small datasets.
- Use a significance test before claiming one method beats another.
- Score supervised methods with cross-validation.
- Compare against the fold-honest
best_single_cvbaseline, not only the oracle-picked best single measure. - State the dataset, its scale, and whether it measures similarity or relatedness.
➡️ Next: Datasets.
- Spearman, C. (1904). The proof and measurement of association between two things. Am. J. Psychology.
- Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30.
- Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2).
- Efron, B., & Tibshirani, R. (1993). An Introduction to the Bootstrap.