Skip to content

Fix critical KV cache crash bug #771 #777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix critical KV cache crash bug #771 #777

wants to merge 1 commit into from
+47 −1

Conversation

@anivar
Copy link

@anivar anivar commented Jul 20, 2025 •
edited
Loading

Problem

Issue #771: Production servers crash with std::length_error: vector during KV cache management.

Root Cause

Integer underflow in update_slots() at line 1714:

slot.cache_tokens.resize(slot.cache_tokens.size() - n_discard);

When n_discard >= cache_tokens.size(), the subtraction underflows, requesting massive memory allocation.

Fix

Added bounds checking before resize:

if (n_discard >= 0 && (size_t)n_discard < slot.cache_tokens.size()) {
    slot.cache_tokens.resize(slot.cache_tokens.size() - n_discard);
} else {
    slot.cache_tokens.clear();
}

Testing

  • Builds successfully
  • Unit test verifies fix handles edge cases
  • No crashes with various n_discard values

Fixes production crashes during high memory pressure scenarios.

@anivar
Fix critical KV cache crash bug causing std::length_error
8083573 
Resolves issue mozilla-ai#771 where server crashes with std::length_error when
KV cache context shifting attempts to resize cache_tokens vector with
integer underflow.

The bug occurs in update_slots() when n_discard >= cache_tokens.size(),
causing cache_tokens.resize(size - n_discard) to underflow and request
massive memory allocation, triggering std::length_error exception.

Changes:
- Add bounds checking before cache_tokens.resize() in server.cpp:1714
- Clear cache_tokens when n_discard would cause underflow
- Prevent negative n_discard values from causing issues

This fix prevents production server crashes reported with Chinese text
translation workloads and high memory pressure scenarios.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@mofosyne mofosyne mofosyne approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

llama.cpp

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@anivar @mofosyne