Re-enable Arm CPU vLLM HUD Benchmarks #120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

fadara01 wants to merge 1 commit into main from reenable-arm-vllm-perf

Collaborator

fadara01 commented Dec 16, 2025 •

edited

Loading

re-enables the benchmarks disabled in Several minor tweak to vllm benchmark workflow #114
builds on Enable aarch64 CPU performance benchmarks vllm-project/vllm#26494
updates benchmarks
uses ON_CPU=1 for both Arm & Intel, which mirrors the corresponding script we call in vLLM.

meta-cla bot added the cla signed label

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Failure

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 temporarily deployed to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Inactive

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

December 16, 2025 12:33

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 requested a deployment to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions In progress

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Inactive

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 had a problem deploying to pytorch-x-vllm

January 8, 2026 10:59

— with

GitHub Actions Error

fadara01 temporarily deployed to pytorch-x-vllm

January 8, 2026 11:51

— with

GitHub Actions Inactive

fadara01 force-pushed the reenable-arm-vllm-perf branch from ab6d64f to 0f7978f Compare

January 8, 2026 12:37


          Re-enable Arm CPU vLLM HUD Benchmarks

72306f2

- re-enables the benchmarks disabled in #114
- related to vllm-project/vllm#26494 (not sure which needs to go in first)
- use default block_size in serving benchmarks (i.e. 128 instead of setting it to 16)

Signed-off-by: Fadi Arafeh <[email protected]>

fadara01 force-pushed the reenable-arm-vllm-perf branch from 0f7978f to 72306f2 Compare

January 8, 2026 12:42

Collaborator Author

fadara01 commented Jan 8, 2026

Hi @huydhn - this is now working. See https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm with device=arm-cpu

Could you please review?
Thank you!

fadara01 requested a review from huydhn

January 8, 2026 13:03

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels