Home

Welcome to the Dysca benchmark!

Jie Zhang*, Zhongqi Wang, Leimeng Qi, Zheng Yuan, Bei Yan, Shiguang Shan, Xilin Chen

*Corresponding Author

🔍 Overview

Abstract - Currently many benchmarks have been proposed to evaluate the perception ability of the Large Vision-Language Models (LVLMs). However, most benchmarks conduct questions by selecting images from existing datasets, resulting in the potential data leakage. Besides, these benchmarks merely focus on evaluating LVLMs on the realistic style images and clean scenarios, leaving the multi-stylized images and noisy scenarios unexplored. In response to these challenges, we propose a dynamic and scalable benchmark named Dysca for evaluating LVLMs by leveraging synthesis images. Specifically, we leverage Stable Diffusion and design a rule-based method to dynamically generate novel images, questions and the corresponding answers. We consider 51 kinds of image styles and evaluate the perception capability in 20 subtasks. Moreover, we conduct evaluations under 4 scenarios (i.e., Clean, Corruption, Print Attacking and Adversarial Attacking) and 3 question types (i.e., Multi-choices, True-or-false and Free-form). Thanks to the generative paradigm, Dysca serves as a scalable benchmark for easily adding new subtasks and scenarios. A total of 24 advanced open-source LVLMs and 2 close-source LVLMs are evaluated on Dysca, revealing the drawbacks of current LVLMs.

📊 Comparison with Existing Benchmarks

Comparisons between existing LVLM benchmarks. '⍻' indicates that the benchmarks include both newly collected images / annotations and images / annotations gathered from existing datasets. '*' The scale of our released benchmark is 617K, however Dysca is able to generate unlimited data to be tested.

Benchmark	#Evaluation Data Scale	#Perceptual Tasks	Automatic Annotation	Collecting from Existing Datasets	Question Type	Automatic Evaluation
LLaVA-Bench	0.15K	-	×	⍻	Free-form	√
MME	2.3K	10	×	⍻	True-or-false	√
LVLM-eHub	-	3	√	×	Free-form	×
tiny-LVLM-eHub	2.1K	3	√	×	Free-form	√
SEED-Bench	19K	8	⍻	×	Multi-choices	√
MMBench	2.9K	12	×	⍻	Multi-choices	√
TouchStone	0.9K	10	×	√	Free-form	√
REFORM-EVAL	50K	7	√	×	Multi-choices	√
MM-BigBench	30K	6	√	×	Multi-choices	√
MM-VET	0.2K	4	⍻	⍻	Free-form	√
MLLM-Bench	0.42K	7	×	⍻	Free-form	√
SEED-Bench2	24K	10	⍻	×	Multi-choices	√
BenchLMM	2.4K	15	×	×	Free-form	√
JourneyDB	5.4K	2	√	√	Free-form, Multi-choices	√
Dysca (Ours)	617K*	20	√	√	Free-form, Multi-choices, True-or-false	√