dataset: Add REAL_MM_RAG benchmark #3224
Conversation
mteb/tasks/Image/Any2AnyRetrieval/eng/RealMMRagBenchRetrieval.py
Outdated
Show resolved
Hide resolved
mteb/tasks/Image/Any2AnyRetrieval/eng/RealMMRagBenchRetrieval.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Roman Solomatin <[email protected]>
|
can you run |
KennethEnevoldsen
left a comment
KennethEnevoldsen
left a comment
There was a problem hiding this comment.
Hi great to see a PR and congratulations on the paper release!
I think the main thing that is missing at the moment is documentation I have put a few pointers below.
Note: will be influenced by #3222 (if merged we can move it down to the retrieval section)
| "RealMMRagTechSlidesRetrieval", | ||
| ], | ||
| ), | ||
| description="Realistic and multi-modal document retrieval benchmark.", |
There was a problem hiding this comment.
This description is too short. Why should I prefer this over another VDR benchmark
| class RealMMRagFinReportRetrieval(AbsTaskAny2AnyRetrieval): | ||
| metadata = TaskMetadata( | ||
| name="RealMMRagFinReportRetrieval", | ||
| description="Retrieve associated pages according to questions.", |
There was a problem hiding this comment.
This description is too vague - It should be clear from the description what queries and corpus it contains, as well as the retrieval goal. Please fix this for all tasks.
There was a problem hiding this comment.
@Samoed should we snake-case the filename (easier to merge with v2)
| "MIEB(Img)", | ||
| "VisualDocumentRetrieval", | ||
| "JinaVDR", | ||
| "REAL_MM_RAG" |
There was a problem hiding this comment.
Let us not add it to a benchmark yet due to #3222 (this means that we can merge this without caring about the other PR and once both is merge then we can add both)
add better descriptions on benchmark subsets
|
@KennethEnevoldsen @Samoed |
There was a problem hiding this comment.
Since v2 release, you need to move your tasks into retrieval/eng/ folder
| from __future__ import annotations | ||
|
|
||
| from datasets import load_dataset | ||
|
|
||
| from mteb.abstasks.Image.AbsTaskAny2AnyRetrieval import AbsTaskAny2AnyRetrieval | ||
| from mteb.abstasks.TaskMetadata import TaskMetadata |
There was a problem hiding this comment.
This also would be
| from __future__ import annotations | |
| from datasets import load_dataset | |
| from mteb.abstasks.Image.AbsTaskAny2AnyRetrieval import AbsTaskAny2AnyRetrieval | |
| from mteb.abstasks.TaskMetadata import TaskMetadata | |
| from datasets import load_dataset | |
| from mteb.abstasks.retrieval import AbsTaskRetrieval | |
| from mteb.abstasks.task_metadata import TaskMetadata |
| "image": None, | ||
| "modality": "text", |
There was a problem hiding this comment.
You shouldn't add columns with None and don't need modality column
| prompt={"query": "Find a screenshot that relevant to the user's question."}, | ||
| descriptive_stats={ | ||
| "n_samples": None, | ||
| "avg_character_length": { | ||
| "test": { | ||
| "average_document_length": 141.5, | ||
| "num_documents": 19, | ||
| "num_queries": 853, | ||
| "average_relevant_docs_per_query": 1.0, | ||
| } | ||
| }, | ||
| }, |
There was a problem hiding this comment.
We don't have descriptive_stats in task metadata. You need to use task.calculate_desriptive_statistics()
There was a problem hiding this comment.
| prompt={"query": "Find a screenshot that relevant to the user's question."}, | |
| descriptive_stats={ | |
| "n_samples": None, | |
| "avg_character_length": { | |
| "test": { | |
| "average_document_length": 141.5, | |
| "num_documents": 19, | |
| "num_queries": 853, | |
| "average_relevant_docs_per_query": 1.0, | |
| } | |
| }, | |
| }, | |
| prompt={"query": "Find a screenshot that relevant to the user's question."}, |
| from mteb.abstasks.TaskMetadata import TaskMetadata | ||
|
|
||
|
|
||
| def _load_data( |
There was a problem hiding this comment.
You can reupload your tasks using task.push_dataset_to_hub() to use our format
|
Can you merge main to resolve conflicts? |
I'm working on it. |
3 participants
I have outlined why this dataset is filling an existing gap in
mtebI have tested that the dataset runs with the
mtebpackage.I have run the following models on the task (adding the results to the pr). These can be run using the
mteb run -m {model_name} -t {task_name}command.ibm-granite/granite-vision-3.3-2b-embeddingjinaai/jina-embeddings-v4I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
I have considered the size of the dataset and reduced it if it is too big (2048 examples is typically large enough for most tasks)