Open
Conversation
Commit xjdr-alt#28 changed `apply_rotary_embed` to have dtype parameter with default float32, and forces attention softmax to be done float32. Since `attention` doesn't specify the dtype parameter when calling `apply_rotary_embed`, and output matmul doesn't convert back from float32 to match the values type, this is an issue if you're running BF16. This specifies the existing xq.dtype for the dtype parameter when calling `apply_rotary_embed` (alternatively, we could cast keys to float32 in `scores = torch.matmul(xq, keys)`), and converts the scores to match values at the output matmul.
Owner
|
@Arrabonae could you take a look |
citizenhicks
approved these changes
Oct 8, 2024
Contributor
citizenhicks
left a comment
citizenhicks
left a comment
There was a problem hiding this comment.
tested this, works well. thanks for spotting this issue!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit #28 changed
apply_rotary_embedto have dtype parameter with default float32, and forces attention softmax to be done float32. Sinceattentiondoesn't specify the dtype parameter when callingapply_rotary_embed, and theoutputmatmul doesn't convert back from float32 to match the values type, this is an issue if you're running BF16.This specifies the existing xq.dtype for the dtype parameter when calling
apply_rotary_embed(alternatively, we could castkeysto float32 inscores = torch.matmul(xq, keys)), and castsscoresto matchvaluesat theoutputmatmul.