DAGE-127: [M2] Add MTEB top model comparison metrics

[nseidan]

• Added MTEB Leaderboard comparison metrics
  Eg.

....
"evaluation_time": 555.2600812911987,
  "kg_co2_emissions": null,
  "mteb_leaderboard_model_comparison": {
    "top_model": "Qwen/Qwen3-Embedding-8B",
    "top_model_avg_main_score": 69.4402,
    "user_model": "sentence-transformers/all-MiniLM-L12-v2",
    "user_model_mteb_avg_main_score": 45.455999999999996,
    "user_model_custom_task_main_score": 50.624,
    "model_main_score_diff": "Your model's main_score on custom task main_score=50.62 vs. MTEB leaderboard shown avg_main_score=45.46",
    "warning": "Your model=sentence-transformers/all-MiniLM-L12-v2 is 27.10% worse than the top model=Qwen/Qwen3-Embedding-8B."
  }

[nicolo-rinaldi]

Just a comment for future work: now we are looking at the English leaderboard. What happens if the dataset is not in English? We should in some way address this comparison

[nseidan]

out tool focuses on only English as of now. We need different ways to handle other languages starting from LLMs.
To your question-> there is a separate multilingual MTEB leaderboard containing 200+ languages.

[nicolo-rinaldi]

Why do we filter on the number of tasks? Are there many models with few tasks executed?

[nseidan]

See the comment. # include "mostly complete" tasks for the models, so excluding models which are run on a few tasks

[nseidan]

FYI: mteb version 2x, added tasks even they run on a few tasks. And the mteb leaderboard shows v.2x mteb verson results.