Skip to content

[Perf][Qwen3-TTS] Vectorize _LocalPredictorKVCache to eliminate per-token loops#1581

Draft
gcanlin wants to merge 1 commit intovllm-project:mainfrom
gcanlin:tts-code-erf
Draft

[Perf][Qwen3-TTS] Vectorize _LocalPredictorKVCache to eliminate per-token loops#1581
gcanlin wants to merge 1 commit intovllm-project:mainfrom
gcanlin:tts-code-erf

Conversation

@gcanlin
Copy link
Contributor

@gcanlin gcanlin commented Mar 1, 2026

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Related to #938.

  • Vectorize block table initialization: replace nested Python for i/for j loop with a broadcasted outer product of two torch.arange tensors
  • Vectorize positions computation in build_attn_metadata: replace per-request torch.arange + list append + torch.cat loop with a 2D grid + boolean mask approach
  • Vectorize slot_mapping computation: replace doubly-nested per-token Python loop (with .item() calls on every token) with bulk torch.repeat_interleave, integer division, and advanced indexing

Test Plan

To be done.

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
  • The test results. Please paste the results comparison before and after, or the e2e results.
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
  • (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

… loops

Signed-off-by: gcanlin <canlinguosdu@gmail.com>
@gcanlin gcanlin changed the title [Perf] Vectorize _LocalPredictorKVCache to eliminate per-token loops [Perf][Qwen3-TTS] Vectorize _LocalPredictorKVCache to eliminate per-token loops Mar 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant