Skip to content

Commit c39b6c4

Browse files
authored
feat: add VLMEvalKit-compatible Qwen task variants for MMMU and MMStar (#1021)
Add new task variants that use VLMEvalKit-style prompt formatting: - mmmu_val_qwen: Uses 'Question: {q}' prefix and 'Answer with the option letter only.' suffix - mmstar_qwen: Uses same VLMEvalKit-compatible prompt structure These variants help users reproduce benchmark scores closer to official Qwen results reported in VLMEvalKit evaluations. Usage: python -m lmms_eval --model qwen2_5_vl --tasks mmmu_val_qwen,mmstar_qwen ... Addresses score reproduction gaps reported in Issues #935, #932, #881, #901
1 parent 9b153d3 commit c39b6c4

File tree

2 files changed

+33
-0
lines changed

2 files changed

+33
-0
lines changed
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
dataset_path: lmms-lab/MMMU
2+
task: "mmmu_val_qwen"
3+
test_split: validation
4+
output_type: generate_until
5+
doc_to_visual: !function utils.mmmu_doc_to_visual
6+
doc_to_text: !function utils.mmmu_doc_to_text
7+
doc_to_target: "answer"
8+
doc_to_messages: !function utils.mmmu_doc_to_messages
9+
process_results: !function utils.mmmu_process_results
10+
11+
metric_list:
12+
- metric: mmmu_acc
13+
aggregation: !function utils.mmmu_aggregate_results
14+
higher_is_better: true
15+
16+
lmms_eval_specific_kwargs:
17+
default:
18+
format: "qwen3_vl"
19+
pre_prompt: "Question: "
20+
post_prompt: "Answer with the option letter only."
21+
open_ended_prompt: "Please answer the question directly."
22+
include: _default_template_yaml
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
dataset_path: Lin-Chen/MMStar
2+
task: "mmstar_qwen"
3+
doc_to_visual: !function utils.mmstar_doc_to_visual
4+
doc_to_text: !function utils.mmstar_doc_to_text
5+
process_results: !function utils.mmstar_process_results
6+
7+
lmms_eval_specific_kwargs:
8+
default:
9+
pre_prompt: "Question: "
10+
post_prompt: "Answer with the option letter only."
11+
include: _default_template_yaml

0 commit comments

Comments
 (0)