Test mmbenchv1.1

[ZhaoyangLi-nju]

how to test mmbenchv1.1?

[kennymckormick]

Hi, @ZhaoyangLi-nju ,

We recommend you to adopt VLMEvalKit: https://github.com/open-compass/VLMEvalKit to evaluate mmbench v1.1. You need to first clone the repo and install it locally.

Usage: https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Quickstart.md

# Multi-GPU
torchrun --nproc-per-node={N_PROC} run.py --data {DATASET_NAME} --model {MODEL_NAME}
# Single-GPU
python run.py --data {DATASET_NAME} --model {MODEL_NAME}

For MMBench v1.1, the corresponding dataset names are:

• MMBench_DEV_EN_V11
• MMBench_DEV_CN_V11
• MMBench_TEST_EN_V11
• MMBench_TEST_CN_V11

For test splits, you need to submit the prediction files to our evaluation service: https://mmbench.opencompass.org.cn/mmbench-submission to get the final scores.

[zjuAJW]

@kennymckormick How to download mmbench v1.1 data?