Block matmul and kv_cache in dynamic quantization #673
Conversation
HolyFalafel
requested review from
adobrzyn,
afierka-intel,
iboiko-habana,
kamil-kaczor,
ksmusz,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski and
xuechendi
as code owners
There was a problem hiding this comment.
Pull request overview
This PR configures dynamic quantization to exclude specific matrix multiplication and key-value cache operations from quantization. The changes prevent quantization of attention mechanism components that may be sensitive to quantization effects.
Key Changes:
- Added a blocklist to the quantization configuration
- Excluded Matmul and KVCache operation types from quantization
- Excluded specific attention-related operations (matmul_qk, matmul_av, k_cache, v_cache) by name
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
✅ CI PassedAll checks passed successfully against the following vllm commit: |
|
Is it for performance reason? |
No, dynamic quantization for KVCache+Matmul is a new feature that we don't want to enable by default |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
Signed-off-by: Danny Semiat <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
✅ CI PassedAll checks passed successfully against the following vllm commit: |
Currently disabling matmul and kv_cache in dynamic quantization mode --------- Signed-off-by: Danny Semiat <[email protected]>
3 participants
Currently disabling matmul and kv_cache in dynamic quantization mode