New IQ2_KT, IQ3_KT and IQ4_KT, V2#529

Merged

ikawrakow merged 33 commits intomainfrom

ik/new_iq2kt_v2

Jun 18, 2025

Commits on Jun 18, 2025

New iq4_kt trellis
Iwan Kawrakow
committed
Something is not working with the AVX2 dot product
Iwan Kawrakow
committed
New iq4_kt: CUDA MMVQ
Iwan Kawrakow
committed
New iq4_kt: CUDA MMQ
Iwan Kawrakow
committed
For now have only iq4_kt use the new trellis
Iwan Kawrakow
committed
Fix iq2_kt that got broken along the way
Iwan Kawrakow
committed
New iq4_kt: AVX2 dot product finally works
Iwan Kawrakow
committed
New iq4_kt: fix vanilla AVX2
Iwan Kawrakow
committed
New iq4_kt: NEON implementation
Iwan Kawrakow
committed
New iq4_kt: slightly faster NEON
Iwan Kawrakow
committed
New iq4_kt: slightly faster NEON
Iwan Kawrakow
committed
New iq4_kt: faster NEON
Iwan Kawrakow
committed
Minor
Iwan Kawrakow
committed
New iq4_kt trellis: not working Metal implementation
Iwan Kawrakow
committed
Remove the extra 4 bytes of row meta data that is no longer used
Iwan Kawrakow
committed
Cleanup
Iwan Kawrakow
committed
Adding forgottent file
Iwan Kawrakow
committed
Switching iq2_kt to new trellis - CUDA MMQ
Iwan Kawrakow
committed
New iq2_kt: CUDA GEMV
Iwan Kawrakow
committed
New iq2_kt: AVX2 dequantize
Iwan Kawrakow
committed
New iq2_kt: AVX2 GEMM/GEMV
Iwan Kawrakow
committed
Adding forgotten file
Iwan Kawrakow
committed
New iq2_kt: NEON GEMM/GEMV
Iwan Kawrakow
committed
New iq2_kt: slightly faster NEON GEMM
Iwan Kawrakow
committed
New iq2_kt: Metal - very slow.
Iwan Kawrakow
committed
Add missing break
Iwan Kawrakow
committed
Trying @louiehelm's multiplier
Iwan Kawrakow
committed
CPU
Iwan Kawrakow
committed
iq3_kt: use integer trellis + CUDA dequantize and MMVQ
Iwan Kawrakow
committed
iq3_kt: MMQ
Iwan Kawrakow
committed
iq3_kt: AVX2 GEMM
Iwan Kawrakow
committed
iq3_kt: AVX2 GEMV
Iwan Kawrakow
committed
The trellis quants now need super-blocks of 256, so we need a check
Iwan Kawrakow
committed