The test results for Qwen3VL show significant discrepancies compared to the official report.

[Fu-Fu-Fu-Fu]

I ran the scripts provided by lmms_eval directly: vllm_qwen3vl.sh or vllm_generate_qwen3vl.sh to test qwen3vl-8b-instruct and thinking, using the tasks videomme, videommmu, and mvbench. I found that the test results differed significantly from the official report, being more than 5%, or even 10%, lower. For example, for videommmu, I ran vllm_qwen3vl.sh, only modifying the model and task. The model used is qwen3vl_8b_thinking.

The official report states 72.8, and even excluding parameters like the maximum frame rate, the difference shouldn't be more than 10%. Have you, the author, encountered this problem before?

[JinXins]

Similar issue, I evaluated the Qwen3-VL-Thinking model on VideoMME, but the results were significantly lower than the results in the official repository. QAQ

[Fu-Fu-Fu-Fu]

@kcz358 请问这个问题怎么解决呀。我看到有issue针对videomme对qwen3vl.py做了修改，但是针对其他数据集呢？其次不用vllm，测试发现特别慢，两张卡对于videommmu都要跑十几个小时

不知道是不是对videomme做过优化，不用vllm，我用4张卡跑videomme速度好一点，大概1-3小时（跑着跑着又预计十几个小时了，好奇怪啊），然而跑videommmu或者mvbench就非常非常慢

accelerate launch --num_processes 4 --main_process_port 12345 -m lmms_eval
--model qwen3_vl
--model_args pretrained=${MODEL},device_map="auto",max_pixels=200704,max_num_frames=16,fps=2,attn_implementation=flash_attention_2
--tasks ${TASKS}
--batch_size 1
--log_samples
--use_cache ./cache/sqlite_cache.db
--log_samples_suffix ${LOG_SUFFIX}
--output_path ${OUTPUT_PATH}
--predict_only