Qualcomm AI Engine Direct - Add MHA2SHA pass #15438
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary:
- Integrated mha into sha pass and implemented it in qnn_preprocess
- Refactored mha in static llama
- Added support for masked softmax
- Included spin quant r3 support
- Combined the n_heads key-value cache into a single cache for each
layer to decrease the number of inputs and outputs, which enhances
performance.
- Deprecated ShiftPointer kv updater mode
- Since each layer now has its own kv cache, the v cache no longer
benefits from ShiftPointer, which previously avoided copying the new
v cache to the input v cache. To prevent user confusion, ShiftPointer
mode has been deprecated
- Applied the correct input template for smollm2 135m
- Corrected the quantization annotation for reshape
- Remove outdated code from CanonicalizeConv
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15438
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 18e7db1 with merge base 3485495 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
|
@pytorchbot label "release notes: qualcomm" |
pytorch-bot
bot
added
the
release notes: qualcomm
|
Hi @cccclai, Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
We observed that quantizing and compiling the original sha model requires a significant amount of time. Switching to the mha model speeds up this process. Therefore, we investigated whether converting the mha model after quantization is feasible. However, we cannot perform this conversion during the to_edge transformation, as splitting the convolution weights to sha would require modifying the state_dict, which is not permitted at that stage. Therefore, we decided to apply this pass during qnn_preprocess.
Summary:
Results
Follow README setting, test on SM8750 with QNN 2.37. Compared the new pass
convert_mha_to_shawith original sha structure