• [2026-01] 🚀 MM-BRIGHT Launch: We release the MM-BRIGHT benchmark, dataset, and evaluation code!
• [2026-01] 🛠️ Code: Full evaluation code for all 4 tasks is released.

Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements—particularly images such as diagrams, charts, and screenshots—that require intensive reasoning to identify relevant documents.

MM-BRIGHT bridges this gap as the first multimodal benchmark for reasoning-intensive retrieval.

MM-BRIGHT evaluates retrieval across four tasks of increasing multimodal complexity:

Finding: Even state-of-the-art models struggle on MM-BRIGHT. BM25 achieves only 8.5 nDCG@10, while the best multimodal model (Nomic-Vision: 27.6) actually underperforms the best text-only model (DiVeR: 32.2).

git clone https://github.com/mm-bright/MM-BRIGHT.git
cd MM-BRIGHT
pip install -r requirements.txt

The dataset is automatically loaded from Hugging Face:

from datasets import load_dataset

# Load documents
docs = load_dataset("mm-bright/MM-BRIGHT", "documents", split="academia")

# Load queries (Task 1/2)
queries = load_dataset("mm-bright/MM-BRIGHT", "examples", split="academia")

# Load multimodal queries (Task 3/4)
mm_queries = load_dataset("mm-bright/MM-BRIGHT", "examples_multimodal", split="academia")

python run_task1.py --dataset_dir . --model bm25 --domains academia biology chemistry

python run_task2.py --dataset_dir . --model nomic-vision --domains academia biology

python run_task3.py --dataset_dir . --model clip --domains academia biology

python run_task4.py --dataset_dir . --model clip --domains academia biology

Use the experiment runner to evaluate all models across all domains:

# Dry run - see all commands
python run_experiments.py --dry_run

# Execute all experiments
python run_experiments.py --dataset_dir .

# Run specific tasks only
python run_experiments.py --dataset_dir . --tasks 1 2

MM-BRIGHT/
├── run_task1.py          # Task 1: Text → Text
├── run_task2.py          # Task 2: Text+Image → Text
├── run_task3.py          # Task 3: Text+Image → Image
├── run_task4.py          # Task 4: Text+Image → Text+Image
├── run_experiments.py    # Batch experiment runner
├── src/
│   ├── data.py           # HuggingFace data loading
│   ├── caching.py        # Embedding cache management
│   ├── eval_runner.py    # Unified evaluation framework
│   ├── utils.py          # Shared utilities
│   ├── models/           # Custom model definitions
│   │   ├── gritlm7b.py
│   │   └── nvmmembed.py
│   └── retrievers/       # Task-specific retrievers
│       ├── task1_text.py
│       ├── task2_multimodal.py
│       ├── task3_image.py
│       └── task4_pair.py
└── outputs/              # Evaluation results

If you use MM-BRIGHT in your work, please cite our paper:

soon

This project is licensed under CC-BY-4.0.

MM-BRIGHT is built on top of the excellent BRIGHT benchmark and extends it to the multimodal domain. We thank the Stack Exchange community for providing the raw data that makes this benchmark possible.