Skip to content

Conversation

@rootfs
Copy link
Collaborator

@rootfs rootfs commented Sep 12, 2025

What type of PR is this?

The current router reason bench is based on MMLU-Pro. This new bench tests more cases with more datasets.

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Release Notes: Yes/No

…tion support. Add ARC, GPQA, TruthfulQA, CommonsenseQA, and HellaSwag datasets with optimized token limits and robust answer extraction.

Signed-off-by: Huamin Chen <[email protected]>
@github-actions
Copy link

github-actions bot commented Sep 12, 2025

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 bench

Owners: @yuezhu1, @Xunzhuo
Files changed:

  • bench/LICENSE
  • bench/MANIFEST.in
  • bench/README.md
  • bench/benchmark_comparison.sh
  • bench/build_and_test.sh
  • bench/comprehensive_bench.sh
  • bench/pyproject.toml
  • bench/requirements.txt
  • bench/setup.py
  • bench/vllm_semantic_router_bench/__init__.py
  • bench/vllm_semantic_router_bench/cli.py
  • bench/vllm_semantic_router_bench/dataset_factory.py
  • bench/vllm_semantic_router_bench/dataset_implementations/__init__.py
  • bench/vllm_semantic_router_bench/dataset_implementations/arc_dataset.py
  • bench/vllm_semantic_router_bench/dataset_implementations/commonsenseqa_dataset.py
  • bench/vllm_semantic_router_bench/dataset_implementations/gpqa_dataset.py
  • bench/vllm_semantic_router_bench/dataset_implementations/hellaswag_dataset.py
  • bench/vllm_semantic_router_bench/dataset_implementations/mmlu_dataset.py
  • bench/vllm_semantic_router_bench/dataset_implementations/truthfulqa_dataset.py
  • bench/vllm_semantic_router_bench/dataset_interface.py
  • bench/vllm_semantic_router_bench/router_reason_bench_multi_dataset.py
  • bench/router_reason_bench.py
  • bench/vllm_semantic_router_bench/bench_plot.py

vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

@netlify
Copy link

netlify bot commented Sep 12, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit a7aafcd
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/68c489c10146a80008f6ada0
😎 Deploy Preview https://deploy-preview-125--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@netlify
Copy link

netlify bot commented Sep 12, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit 6070f9c
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/68c6cf0afa504b0008848e69
😎 Deploy Preview https://deploy-preview-125--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Signed-off-by: Huamin Chen <[email protected]>
@rootfs rootfs merged commit d3c6d91 into vllm-project:main Sep 14, 2025
9 checks passed
@rootfs rootfs mentioned this pull request Sep 23, 2025
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants