Skip to content

exercises.md

Latest commit

 

History

History
89 lines (61 loc) · 3.5 KB

exercises.md

File metadata and controls

89 lines (61 loc) · 3.5 KB
Raw

Exercises

Hands-on problems that build from "run the code" to "do small research." Each has a hint; solutions are intentionally left open so they work as coursework. Most run offline; a few need datasets.load(...) (one-time download) or WordNet.

Setup: pip install -r requirements.txt and, for WordNet exercises, pip install nltk && python -m similarity_ensemble.datasets --nltk.

A. Warm-up (understand the pieces)

A1. Compute all six lexical measures for the pairs (car, automobile), (gem, jewel), (noon, string). Which measure gives the highest score to the synonym pairs? Which gives the lowest to the unrelated pair? Hint: measures.build_feature_table(pairs, measures.LEXICAL_MEASURES).

A2. For MC-30, compute the Spearman correlation of each base lexical measure with the human gold. Are lexical measures good at synonymy? Explain why/why not using two example pairs. Hint: datasets.load("mc30"), then metrics.spearman.

A3. Plot the inter-measure correlation heatmap for MC-30. Which two measures are the most redundant? Which pair is the most diverse? Hint: viz.measure_correlation_heatmap.

B. Ensembling

B1. On RG-65, compare MeanEnsemble, MedianEnsemble, LinearStacking, and GAWeightedEnsemble. Report Spearman with 95% bootstrap CIs. Does any ensemble significantly beat the best single measure (Steiger–Williams test)? Hint: evaluation.benchmark + metrics.bootstrap_ci + metrics.steiger_williams_test.

B2. Add the WordNet measures (path, wup, lin) to the base pool and repeat B1. Does adding a different family change the ensemble gain? Relate your answer to "diversity."

B3. The genetic ensemble exposes ga.weights. After fitting on RG-65, which measures receive near-zero weight? What does that tell you about the pool?

C. Methodology (think like a reviewer)

C1. Fit LinearStacking and evaluate it (i) on the training pairs and (ii) with 5-fold cross-validation. How large is the optimism gap? Why does it matter on a 30-pair dataset?

C2. Replace MC-30 with SimLex-999. Do the same measures rank differently? Explain using the similarity-vs-relatedness distinction.

C3. The genetic ensemble optimizes Spearman; LinearStacking minimizes squared error. Construct (or find) a case where the GA wins on Spearman but loses on RMSE. What does this say about choosing an objective?

D. Extend the platform (mini-projects)

D1. Implement a new base measure (e.g., cosine over character-trigram counts or a Jaro–Winkler ratio). Add a bounds/identity test. Does it improve any ensemble?

D2. Add a dataset downloader for MEN or SimVerb-3500 (a DatasetSpec with citation + mirror URLs). Run the full benchmark on it.

D3. Implement a simple fuzzy ensemble (three membership functions per measure + a small rule base) and compare it to the algebraic and regression baselines. Where does interpretability help?

D4. Add a boosting-style ensemble (sequentially reweight pairs the current ensemble gets wrong). Does it beat plain blending? Report with CIs.

E. Reflection

E1. In ≤200 words: when would you not bother with an ensemble for a semantic similarity task? Give a concrete example.

E2. Design (on paper) a fair benchmark protocol for comparing five similarity systems across accuracy, runtime, and robustness. Which metric is primary, and why?

Want feedback on a solution? Open a GitHub Discussion or a PR adding your notebook to notebooks/community/.