Fix kq_scale for the attention layers of PLaMo2 #14892
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CISC
approved these changes
Jul 26, 2025
CISC
reviewed
Jul 26, 2025
| ggml_view_1d(ctx0, ssm_states_all, d_state*d_inner*n_seqs, | ||
| kv_head*d_state*d_inner*ggml_element_size(ssm_states_all)))); | ||
| ggml_view_1d(ctx0, y_ssm, n_heads*head_dim*d_state*n_seqs, n_heads*head_dim*n_seq_tokens*n_seqs*ggml_element_size(y_ssm)), | ||
| ggml_view_1d(ctx0, ssm_states_all, n_heads*head_dim*d_state*n_seqs, kv_head*n_seqs*n_heads*head_dim*d_state*ggml_element_size(ssm_states_all)))); |
There was a problem hiding this comment.
Mind commenting on these changes?
There was a problem hiding this comment.
These changes actually replace d_inner with n_heads*head_dim for both of y_ssm and ssm_states_all. This is because I thought it's more natural because the ssm state created in get_ssm_rows() has the shape of (d_state , head_dim, n_heads, n_seqs) here:
Line 16359 in 446595b
But, if d_inner is always the same as n_heads*head_dim, I'm happy to revert this change cause it's unnecessary.
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the scale given to
build_attn()in the PLaMo2 model part.It should be
1.0f/sqrtf(float(n_embd_head_v))but it was just1.0and significantly reduces the output quality of the model.This PR also fixes some default values used in
convert_hf_to_gguf.pyfor PLaMo2 to align them with the values used as the default values in themodeling_plamo.pyof PLaMo2.