Open
Conversation
yangulei
requested review from
adobrzyn,
afierka-intel,
iboiko-habana,
kamil-kaczor,
ksmusz,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski and
xuechendi
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug where preempted prompts were incorrectly classified as decode requests, causing runtime errors. The fix introduces a dedicated method to properly identify prompt requests by considering both computed tokens and scheduled tokens, which is especially important for handling preempted sequences.
Changes:
- Added
_is_prompt()method to accurately determine if a request is a prompt, accounting for preempted prompts - Refactored prompt/decode classification logic to use the new helper method
- Added preemption handling test to CI
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| vllm_gaudi/v1/worker/hpu_model_runner.py | Implements _is_prompt() helper method and refactors prompt/decode classification logic to fix preempted prompt handling |
| tests/full_tests/preemption.py | Adds new test file to verify preemption handling with high memory pressure conditions |
| tests/full_tests/ci_gsm8k_tests.sh | Adds preemption test function to CI test suite |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
yangulei
commented
Jan 16, 2026
yangulei
force-pushed
the
preempted_prompt
branch
4 times, most recently
from
b0f4a6e to
06b9b4a
Compare
✅ CI PassedAll checks passed successfully against the following vllm commit: |
yangulei
force-pushed
the
preempted_prompt
branch
from
f414a28 to
a456bef
Compare
✅ CI PassedAll checks passed successfully against the following vllm commit: |
yangulei
force-pushed
the
preempted_prompt
branch
2 times, most recently
from
6bcc307 to
2b0f48c
Compare
Merged
The preempted prompts might failed to mitch the `num_computed_tokens < num_prompt_tokens` test and be treated as decoding then cause runtime error. - add `_is_prompt()` to check if a request is prompt or not. - consider the `num_scheduled_tokens` to handle the preempted prompts. - add test for preemption handling to the CI. --------- Signed-off-by: Youlei Yang <youlei.yang@intel.com>
yangulei
force-pushed
the
preempted_prompt
branch
2 times, most recently
from
a86d05d to
f1bf91d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The preempted prompts might failed to meet the
num_computed_tokens < num_prompt_tokenstest and be treated as decoding then cause runtime error.Changes
_is_prompt()to check if a request is prompt or not.num_scheduled_tokensto handle the preempted prompts.