Update llama.py - Fix embedding generation error #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…in Gemma3ChatHandler
Replace llama_kv_cache_clear -> llama_kv_self_clear
There was a problem hiding this comment.
Pull Request Overview
This PR fixes an embedding generation error by replacing calls to the wrong cache-clear function with the correct one.
- Calls to
llama_kv_cache_clearhave been updated tollama_kv_self_clearin the embedding flow. - Ensures the context cache is properly cleared before and after decoding batches.
Comments suppressed due to low confidence (1)
llama_cpp/llama.py:982
- Add or update unit tests for the
embedfunction to verify that embeddings are generated correctly withllama_kv_self_clearand that the cache is fully cleared before and after decoding.
data: Union[List[List[float]], List[List[List[float]]]] = []
| # decode and fetch embeddings | ||
| data: Union[List[List[float]], List[List[List[float]]]] = [] | ||
|
|
||
| def decode_batch(seq_sizes: List[int]): |
There was a problem hiding this comment.
[nitpick] Consider adding a brief comment explaining why llama_kv_self_clear is used here instead of the previous llama_kv_cache_clear, to clarify the intended cache-clearing behavior for future maintainers.
Suggested change
| def decode_batch(seq_sizes: List[int]): | |
| def decode_batch(seq_sizes: List[int]): | |
| # Clear the self-attention key-value cache to prepare for decoding the next batch. | |
| # `llama_kv_self_clear` is used here instead of `llama_kv_cache_clear` because it specifically | |
| # clears the cache for self-attention mechanisms, which is required for accurate embedding generation. |
Copilot uses AI. Check for mistakes.
|
We can close this PR if |
okaris
force-pushed
the
main
branch
2 times, most recently
from
ef28569 to
a096d51
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replace llama_kv_cache_clear -> llama_kv_self_clear.
Revert back until llama_kv_cache_clear function will be fixed