Skip to content

Conversation

@milistu
Copy link
Contributor

@milistu milistu commented Dec 2, 2025

Description

This PR extends NanoBEIREvaluator to support custom HuggingFace NanoBEIR dataset paths, enabling evaluation on alternative NanoBEIR collections like:

Motivation

This feature was developed while creating an embedding model for the Serbian language. During this process, I noticed that under-resourced languages often lack evaluation benchmarks, making it difficult for practitioners to compare and select models for their applications.

By enabling NanoBEIREvaluator to accept custom dataset paths, researchers and developers can now:

  • Evaluate models on translated NanoBEIR datasets
  • Create and share benchmarks for their own languages
  • Compare multilingual models on localised evaluation sets

I plan to publish my findings and a guide on translation and fine-tuning embedding models for low-resource languages in the near future.

Changes

NanoBEIREvaluator

  • Accept custom HuggingFace dataset paths (e.g., "sentence-transformers/NanoClimateFEVER-bm25")
  • Support "relevance" subset in addition to "qrels" for newer dataset formats
  • Support positive-corpus-ids (list) in addition to corpus-id (single value)
  • Validate custom paths follow NanoBEIR naming convention

Tests

  • Updated existing validation test for the new error message
  • Added tests for custom path validation

Usage Example

# Predefined datasets (existing behaviour)
evaluator = NanoBEIREvaluator(dataset_names=["climatefever", "msmarco"])

# Custom HuggingFace paths (new feature)
evaluator = NanoBEIREvaluator(
    dataset_names=["sentence-transformers/NanoClimateFEVER-bm25"]
)

@milistu milistu changed the title Feat/nanobeir Extend NanoBEIREvaluator to support custom HuggingFace NanoBEIR datasets Dec 3, 2025
@milistu milistu changed the title Extend NanoBEIREvaluator to support custom HuggingFace NanoBEIR datasets Extend NanoBEIREvaluator to support custom NanoBEIR datasets Dec 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant