Skip to content

[Bugfix] fix kernel error for qwen3-omni #1602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hsliuustc0106 merged 1 commit into from
Mar 2, 2026
+9 −1
Merged

[Bugfix] fix kernel error for qwen3-omni #1602

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion vllm_omni/model_executor/stage_configs/qwen3_omni_moe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ stage_args:
stage_type: llm # Use llm stage type to launch OmniLLM
runtime:
devices: "1"
max_batch_size: 64
max_batch_size: 32
Copy link
Collaborator

@hsliuustc0106 hsliuustc0106 Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we need to change config?

Copy link
Contributor Author

@R2-Y R2-Y Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we only use two cards now, batch size = 64 for code2wav will OOM for qwen3-omni convolution computation

engine_args:
model_stage: code2wav
model_arch: Qwen3OmniMoeForConditionalGeneration
Expand Down
8 changes: 8 additions & 0 deletions vllm_omni/worker/gpu_ar_model_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,14 @@ def sample_tokens(
if grammar_output is not None:
apply_grammar_bitmask(scheduler_output, grammar_output, self.input_batch, logits)

# Correct padding values of prompt_token_ids to match the logits vocabulary size
if logits is not None and not self.input_batch.sampling_metadata.no_penalties:
smd = self.input_batch.sampling_metadata
if smd.prompt_token_ids is not None:
logits_vocab = logits.shape[-1]
if self.input_batch.vocab_size > logits_vocab:
smd.prompt_token_ids = smd.prompt_token_ids.clamp(max=logits_vocab)

with record_function_or_nullcontext("gpu_model_runner: sample"):
sampler_output = self._sample(logits, spec_decode_metadata)

Expand Down