Q8_KV: 8-bit quantization type targeting the KV cache#208

Merged

ikawrakow merged 15 commits intomainfrom

Feb 19, 2025

Commits on Feb 19, 2025

Adding q8_KV - Basics + AVX2 gemm/gemv
Iwan Kawrakow
committed
q8_KV: Better AVX2 gemm
Iwan Kawrakow
committed
q8_KV: Better Zen4 gemm
Iwan Kawrakow
committed
q8_KV: AVX2 gemm/gemv
Iwan Kawrakow
committed
q8_KV: be able to use it for K cache
Iwan Kawrakow
committed
q8_KV: be able to use it for K cache in FA
Iwan Kawrakow
committed
q8_KV: repack it for K*Q in FA
Iwan Kawrakow
committed
q8_KV: slightly faster gemv on Zen4
Iwan Kawrakow
committed
q8_KV: slightly faster gemv on Zen4
Iwan Kawrakow
committed
q8_KV: ARM_NEON
Iwan Kawrakow
committed
q8_KV: use it in FA on NEON
Iwan Kawrakow
committed
q8_KV_r8 - repacked q8_KV
Iwan Kawrakow
committed
q8_KV_r8: don't use nrc_y = 16 on Zen4
Iwan Kawrakow
committed
q8_KV: nrc_y = 16 also doesn't pay off in FA
Iwan Kawrakow
committed
Minor
Iwan Kawrakow
committed