Skip to content

[Perf] Early return in KVCacheManager.allocate_slots #29206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+17 −10
Open

[Perf] Early return in KVCacheManager.allocate_slots #29206

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions vllm/v1/core/kv_cache_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,16 +267,6 @@ def allocate_slots(
else:
new_computed_block_list = self.empty_kv_cache_blocks.blocks

# Free the blocks that are skipped during the attention computation
# (e.g., tokens outside the sliding window).
# We can do this even if we cannot schedule this request due to
# insufficient free blocks.
# Should call this function before allocating new blocks to reduce
# the number of evicted blocks.
self.coordinator.remove_skipped_blocks(
request.request_id, request.num_computed_tokens
)

# The number of computed tokens is the number of computed tokens plus
# the new prefix caching hits
num_computed_tokens = request.num_computed_tokens + num_new_computed_tokens
Expand All @@ -292,6 +282,23 @@ def allocate_slots(
num_encoder_tokens=num_encoder_tokens,
)

if (
num_blocks_to_allocate == 0
and new_computed_block_list is self.empty_kv_cache_blocks.blocks
Copy link
Collaborator

@heheda12345 heheda12345 Nov 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
and new_computed_block_list is self.empty_kv_cache_blocks.blocks
and new_computed_blocks is None

prefer this slightly.

And do we need to add num_computed_tokens > request.num_prompt_tokens to make remove_skipped_blocks to be called in the first decode step? This can help to free the prefill tokens used by the last prefill step but is outside the sliding window of the first decode step. Would be grateful if you can try gpt-oss and gemma3, two models with small sliding window size.

Copy link
Collaborator

@heheda12345 heheda12345 Nov 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And can you add a comment for this may delay remove_skipped_blocks and cache_blocks and give some analysis about it is fine.

):
# Early return as no new blocks needed to be allocated
return self.empty_kv_cache_blocks
Comment on lines +285 to +290
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Skipped sliding-window cleanup when no new blocks allocated

allocate_slots now returns early when num_blocks_to_allocate==0 and no prefix-cache hits, so coordinator.remove_skipped_blocks() is never called on those steps. For sliding-window/ChunkedLocal attention this call is what frees blocks that have fallen outside the window; skipping it leaves those blocks held until the request hits a block boundary or finishes, inflating block_pool usage and causing unnecessary evictions or scheduling failures for long prompts that keep generating within an existing block.

Useful? React with 👍 / 👎.


# Free the blocks that are skipped during the attention computation
# (e.g., tokens outside the sliding window).
# We can do this even if we cannot schedule this request due to
# insufficient free blocks.
# Should call this function before allocating new blocks to reduce
# the number of evicted blocks.
self.coordinator.remove_skipped_blocks(
request.request_id, request.num_computed_tokens
)

if num_blocks_to_allocate > self.block_pool.get_num_free_blocks():
# Cannot allocate new blocks
return None
Expand Down