Support for ViDoRe Benchmark V3

[StupidBuluchacha]

Hi MMEB Team,

I noticed that Vidore-v3 benchmark has been released recently, which is claimed as the most diverse visual document retrieval benchmark to date for enterprise applications. Based on the description of its blog, the benchmark draws from 10 challenging, real-world datasets spanning diverse industrial domains, with 8 publicly released and 2 kept private. The two private subsets will be managed by the MTEB team to ensure benchmark integrity and mitigate overfitting.

We believe MMEB is also one of the most comprehensive benchmarks for multimodal retrieval tasks (including visual document retrieval), thus we wonder if you will integrate vidore-v3 in the near future? If so, would you also manage the private sets like MTEB team? Look forward to your reply sincerely.

[XMHZZ2018]

@StupidBuluchacha

Thanks for your suggestion! Since ViDoRe-v3 follows the same format as the previous versions, it should be quite easy to add — it can directly use the existing data parser from ViDoRe-v1/v2 (https://github.com/TIGER-AI-Lab/VLM2Vec/blob/main/src/data/eval_dataset/vidore_dataset.py), and simply updating the config file should work.

At the moment, there are some plans to incorporate more diverse tasks, including ViDoRe-v3 and a few other ones. However, this depends on the team members’ bandwidth, so there isn’t a concrete timeline yet. Community contributions are, of course, very welcome!

Private datasets are definitely valuable and important, and we may consider supporting them in the future.