[V1] [Hybrid] Enable Full CUDA graph by default for models with mamba2 layers in V1

[tdoublep]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

The mamba2 layer has a lot of CPU overhead which mean its performance is really poor for small batches unless CUDA graphs are used. This means that the default V1 performance is still significantly worse than the default V0 performance (see below).

We recently merged #21401 which solves this problem by enabling full CUDA graphs for decode-only batches. This PR proposes to enable this by default for models with mamba2. Without this change, users will continue to experience poor performance when switching from V0 to V1, unless they really know what they are doing.

In my view, this PR and #21549 would be the last steps required to enable V1 by default for these models and to start stripping out the V0 code.

cc @heheda12345 @tlrmchlsmth

Test Plan

• default V0 performance (main)
• default V1 performance (main)
• default V1 performance (this PR)

Test Result

Default V0 performance (main):

python benchmark_latency.py \
	--model ibm-granite/granite-4.0-tiny-preview \
	--input-len 31500 \
	--output-len 512 \
	--batch-size 1	\
	--num-iters-warmup 3 \
	--num-iters 3 

Avg latency: 3.505638455351194 seconds
10% percentile latency: 3.501975697884336 seconds
25% percentile latency: 3.5030823255656287 seconds
50% percentile latency: 3.5049267050344497 seconds
75% percentile latency: 3.5078387099783868 seconds
90% percentile latency: 3.509585912944749 seconds
99% percentile latency: 3.5106342347245665 seconds

Default V1 performance (main):

VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER python benchmark_latency.py \
	--model ibm-granite/granite-4.0-tiny-preview \
	--input-len 31500 \
	--output-len 512 \
	--batch-size 1	\
	--num-iters-warmup 3 \
	--num-iters 3 

Avg latency: 8.384082738310099 seconds
10% percentile latency: 8.35280047832057 seconds
25% percentile latency: 8.372324891504832 seconds
50% percentile latency: 8.40486558014527 seconds
75% percentile latency: 8.406232006032951 seconds
90% percentile latency: 8.40705186156556 seconds
99% percentile latency: 8.407543774885125 seconds

Default V1 performance (this PR):

VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER python benchmark_latency.py \
	--model ibm-granite/granite-4.0-tiny-preview \
	--input-len 31500 \
	--output-len 512 \
	--batch-size 1	\
	--num-iters-warmup 3 \
	--num-iters 3 

Avg latency: 3.398241866302366 seconds
10% percentile latency: 3.3959803130012007 seconds
25% percentile latency: 3.397328175487928 seconds
50% percentile latency: 3.3995746129658073 seconds
75% percentile latency: 3.3998219304485247 seconds
90% percentile latency: 3.399970320938155 seconds
99% percentile latency: 3.4000593552319334 seconds

(Optional) Documentation Update

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request enables full CUDA graphs by default for hybrid models to improve performance, which is a great optimization. I have one suggestion regarding the implementation to ensure user configurations are not unintentionally overridden.

[mgoin]

Looks reasonable to me, the difference is huge! The main concern would be if we enable full cudagraph by default for Mamba2 but one of the non-mamba layers needs to use an attention backend without full cg support

[mgoin]

Might be worth leaving an info_once log here about what is going on and why

[heheda12345]

Amazing speedup! Can you update the doc?

[heheda12345]

what about maintaining a white list in HybridAttentionMambaModelConfig for models that should use v1? For example I think we can enable mamba1 and minimax soon. has_mamba2 is only a very temporary way to determine cuda graph.

[hmellor]

Why change this default?

Also FYI this config is to be replaced with cudagraph_mode in https://github.com/vllm-project/vllm/pull/20059/files#diff-19413af30c8d2ba8cdaf4989476312ece57b0e9811b39a62286623b99c7fd0a8

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tdoublep.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Josephasafg]

@tdoublep This is great. Suggestion - Everything here is not very mamba2 specific, meaning, once we add full cuda support to mamba1 (we have the PR waiting to be opened), we would want to add this code to it as well without duplicating it. So perhaps its worth removing the 2 suffix from the names here, so we can just take it as is once FCG is supported. wdyt?