Skip to content

Conversation

weireweire
Copy link
Contributor

@weireweire weireweire commented Aug 25, 2025

Purpose

Update flashinfer to

Test Plan

lm_eval on llama3

gpt-oss/eval test on gpt-oss

  • python3 -m gpt_oss.evals --sampler chat_completions --model gpt-oss-120b --reasoning-effort low,medium --n-threads 512 --eval gpqa

Test Result

llama3 FP8 tp2

Tasks Version Filter n-shot Metric Value Stderr
gsm8k 3 flexible-extract 5 exact_match 0.938 ± 0.0108
strict-match 5 exact_match 0.904 ± 0.0132

llama3 FP4 tp1

Tasks Version Filter n-shot Metric Value Stderr
gsm8k 3 flexible-extract 5 exact_match 0.930 ± 0.0114
strict-match 5 exact_match 0.842 ± 0.0163

gpt-oss tp1:
[{'eval_name': 'gpqa', 'model_name': 'gpt-oss-120b-low_temp1.0_20250825_021150', 'metric': 0.6414141414141414}, {'eval_name': 'gpqa', 'model_name': 'gpt-oss-120b-medium_temp1.0_20250825_021150', 'metric': 0.711489898989899}]

(Optional) Documentation Update


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

IwakuraRein and others added 11 commits August 25, 2025 06:24
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: Siyuan Fu <[email protected]>
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates Flashinfer to version 0.2.14.post1, which addresses a performance issue in the allreduce fusion kernel and incorporates API changes. The changes also include enabling FlashInfer autotuning before CUDA graph capture for better performance. My review found a critical typo in a variable name (max_captute_size instead of max_capture_size) in mxfp4.py which would lead to a runtime error. I've provided suggestions to fix this.

@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 25, 2025
mgoin and others added 2 commits August 25, 2025 11:08
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Michael Goin <[email protected]>
Copy link
Collaborator

@yewentao256 yewentao256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work!
Could you also report vllm bench metric results so that we can see if we have some improvement for E2E throughput?

Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for putting everything together. Let's see the CI

@simon-mo simon-mo merged commit ae06788 into vllm-project:main Aug 26, 2025
64 of 70 checks passed
tc-mb pushed a commit to tc-mb/vllm that referenced this pull request Aug 27, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: tc-mb <[email protected]>
tc-mb pushed a commit to tc-mb/vllm that referenced this pull request Aug 27, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: tc-mb <[email protected]>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
xiao-llm pushed a commit to xiao-llm/vllm that referenced this pull request Aug 28, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Xiao Yu <[email protected]>
xiao-llm pushed a commit to xiao-llm/vllm that referenced this pull request Aug 28, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Xiao Yu <[email protected]>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
dumb0002 pushed a commit to dumb0002/vllm that referenced this pull request Aug 28, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2015aroras pushed a commit to 2015aroras/vllm that referenced this pull request Aug 29, 2025
Signed-off-by: Siyuan Fu <[email protected]>
Signed-off-by: siyuanf <[email protected]>
Signed-off-by: Weiliang Liu <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Co-authored-by: Siyuan Fu <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci/build ready ONLY add when PR is ready to merge/full CI is needed v1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants