RoPE: fix back, CUDA support for back + noncont. #11240
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
testing
ggerganov
approved these changes
Jan 14, 2025
Member
ggerganov
left a comment
ggerganov
left a comment
There was a problem hiding this comment.
Now that non-contiguous rope is supported, the extra ggml_cont in build_minicpm3 and build_deepseek2 could likely be removed:
But it would be good to test these models just in case.
If you don't have the models handy, you can simply update the TODO comments with one that asks to perform the test, so that somebody else can do it in the future.
ggerganov
reviewed
Jan 15, 2025
Member
There was a problem hiding this comment.
The build_minicpm3 has the same comments that should be updated
JohannesGaessler
force-pushed
the
rope-backward
branch
from
aaa1243 to
c87818e
Compare
Nexesenex
added a commit
to Nexesenex/croco.cpp
that referenced
this pull request
Jan 15, 2025
Credits : @ Johannes Gaessler
tinglou
pushed a commit
to tinglou/llama.cpp
that referenced
this pull request
Feb 13, 2025
* RoPE: fix back, CUDA support for back + noncont. * fix comments reg. non-cont. RoPE support [no-ci]
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Feb 26, 2025
* RoPE: fix back, CUDA support for back + noncont. * fix comments reg. non-cont. RoPE support [no-ci]
mglambda
pushed a commit
to mglambda/llama.cpp
that referenced
this pull request
Mar 8, 2025
* RoPE: fix back, CUDA support for back + noncont. * fix comments reg. non-cont. RoPE support [no-ci]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the backward pass for RoPE. On master
test-backend-ops gradis failing on a related assert. The backwards pass can be constructed relatively simply by just creating a tensor for the forward pass and then changing the op fromGGML_ROPEtoGGML_ROPE_BACK. One could maybe setggml_tensor.op_paramsinstead of the op but I don't think that would reduce the overall complexity.This PR also adds CUDA support for the RoPE backwards pass and for non-contiguous inputs. The latter is needed for the backwards pass of the KV cache. I also added
__restrict__andconstwhere applicable and simplified the templating a bit.Implicitly
test-backend-opsis already testingGGML_OP_ROPE_BACKvia gradients, I also added an explicit test.