[Feature]: Multimodal Benchmarking Support (MMLM)

[knlnguyen1802]

🚀 The feature, motivation and pitch

1. Motivation

vLLM’s built-in benchmark currently reports

• TTFT – Time-to-First-Token
• TPT – Tokens-Per-Second
• ITL – Inter-Token-Latency

Those latency-centric numbers are perfect for pure-text LLMs, but they do not capture the quality or unique execution characteristics of multimodal large models (MMLMs) that take both text and images. Adding a fit-for-purpose multimodal benchmark would make vLLM even more valuable for researchers and practitioners.

---

2. What is missing right now?

1. Dataset – No out-of-the-box multimodal test set that exercises image → text or text → image abilities.
2. Metrics – Current numbers show speed only; they don’t answer “How well is the model performing on the task?”
3. Evaluation harness – vLLM lacks a driver that loads multimodal samples, feeds them, and aggregates both quality and latency into one report.

---

3. Proposed Solution

3.1 Candidate Benchmark Datasets

Category	Suggested dataset	License	Rationale
VQA	VQAv2, OK-VQA	CC-BY-4.0	Classic image-to-text Q&A
Caption	MS-COCO Captions	CC-BY-4.0	Widely used; automatic metrics available
Reasoning	MMMU, MMBench, ScienceQA	CC-BY-NC-SA	Tests multi-hop visual + text reasoning
Random	ImageNet-1k 5 k random subset	CC BY	Stress-tests generic vision encoder paths

(One small, redistributable subset per task is usually enough—e.g., ~2 k images total.)

3.2 Suggested Metrics

• ETI – Encode-to-Token Interval: wall-time from first image byte received to first generated token.
• FPS-Enc – images processed per second (encoder throughput).
• Continue to report TTFT, TPT, ITL for apples-to-apples comparison with text-only runs

Noted: Benchmark for speculative decoding is available here and support custom dataset but not friendly
https://docs.google.com/document/d/1SbAnLNfCp04lHLJ_cF22IYc_StJ_U3jRRUSc3dQTxO0/edit?pli=1&tab=t.0

cc @DarkLight1337

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

We already support benchmarking multi-modal datasets via HF datasets. We usually use VisionArena for this, e.g.:

python benchmarks/benchmark_serving.py \
  --backend openai-chat \
  --model Qwen/Qwen2.5-VL-3B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --hf-split train \
  --num-prompts 500 \
  --request-rate inf

Regarding the quality of output, that is already covered by our correctness tests which compare our output against HF Transformers library. So, our benchmarking is solely focused on speed (I believe this is also the case for our text models).

Also cc @ywang96 @mgoin

[knlnguyen1802]

We already support benchmarking multi-modal datasets via HF datasets. We usually use VisionArena for this, e.g.:

python benchmarks/benchmark_serving.py \
  --backend openai-chat \
  --model Qwen/Qwen2.5-VL-3B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --hf-split train \
  --num-prompts 500 \
  --request-rate inf

Regarding the quality of output, that is already covered by our correctness tests which compare our output against HF Transformers library. So, our benchmarking is solely focused on speed (I believe this is also the case for our text models).

Also cc @ywang96 @mgoin

Thank you for the response.
Because in MMLM model, the encoder performance is also important, so I want to have that data too and if can support an random dataset, it will be light weight and faster on testing

[DarkLight1337]

We currently only support streaming and E2E metrics as they can be computed from the timing of the API responses without having to modify vLLM code. To measure the encoder performance directly, we need to time the code inside vLLM itself, which is quite intrusive. I suggest using a profiler to determine the relative overhead of the encoder.

[DarkLight1337]

Another way to do this is to record new metrics (https://docs.vllm.ai/en/stable/design/v1/metrics.html). However based on the discussion of #16478, it seems that other maintainers want to limit the set of metrics to reduce overhead.

[knlnguyen1802]

@DarkLight1337 I want to redo #16478 but it will record Multi Modal cache hit only (Number of images, audio that hit the cache) after support #21113. It will help to see similar log like prefix cache, What is your opinion ?

[DarkLight1337]

Sure, that would be great!

[khuonglm]

@DarkLight1337 I am also currently using provided profiling method. Although we could extract information like TTFT, E2E, it is too coarse-grained. When benchmarking MMLM, I want to find the bottleneck of the end-to-end execution, therefore encoder execution time and queuing time are important in my opinion. Therefore, I want to ask if it is a good idea to return that additional information after running profiling.

[h-brenoskuk]

@knlnguyen1802 Don't know if pertinent but I am extending the RandomDataset to RandomMultiModalDataset so that we can have easy to get synthetic data.

After the synth data is operational, I am also looking into improving the benchmarks of multimodal models. Have you advanced on this front?

[knlnguyen1802]

@h-brenoskuk I do not plan on it and it sounds great. Grateful to see your contribution on it.

[DarkLight1337]

cc @njhill @robertgshaw2-redhat are you ok with adding these metrics?

[DarkLight1337]

cc @markmc