Evaluate reranker models

[davispuh]

Hi!

I'm trying to find best open source multilingual rerarenker model but it looks like MTEB contains only embedding models?
But I see there is Reranking score so would it be possible to also add rerarenker models?

For example, there are:

• Qwen3-Reranker-8B
• mxbai-rerank-large-v2
• jina-reranker-v3
• zerank-1
• granite-embedding-reranker-english-r2 (English only)

[Samoed]

Yes, but for now we have retrieval tasks only benchmark (RTEB, BEIR, etc), but rerankers required reranking only tasks, and we need to create new benchmark for this.