Much faster CPU prompt processing (part 1)#531

Merged

ikawrakow merged 24 commits intomainfrom

Jun 17, 2025

Commits on Jun 13, 2025

q6_K dequantizing GEMM
Iwan Kawrakow
committed
Much easier: just use different vec_dot types!
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed

Commits on Jun 14, 2025

Finally q6_K x q8_2_x4 dot product works
Iwan Kawrakow
committed
Very slightly better
Iwan Kawrakow
committed
We don't need the changes in ggml.c
Iwan Kawrakow
committed
Fix AVX2
Iwan Kawrakow
committed
iq2_xs
Iwan Kawrakow
committed
Fix AVX2
Iwan Kawrakow
committed
iq2_s
Iwan Kawrakow
committed
q3_K
Iwan Kawrakow
committed

Commits on Jun 15, 2025

Fix q8_k_r8 on Zen4
Iwan Kawrakow
committed
q3_K: repack to q8_k_r8 instead of q8_0_r8
Iwan Kawrakow
committed
q3_K: don't scale when all quants in a block are <= 127 when repacking
Iwan Kawrakow
committed
iq2_s: repack to q8_k_r8 instead of q8_0_r8
Iwan Kawrakow
committed
iq2_xs: rapck to q8_k_r8
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
iq2_xs: repack to q8_k_r8
Iwan Kawrakow
committed
iq3_xxs: repack to q8_k_r8
Iwan Kawrakow
committed
iq3_s: use q8_k_r8
Iwan Kawrakow
committed
iq1_s: repack to q8_k_r8
Iwan Kawrakow
committed
iq1_m: repack to q8_k_r8
Iwan Kawrakow
committed

Commits on Jun 16, 2025

iq1_m: slightly faster
Iwan Kawrakow
committed
Slightly faster
Iwan Kawrakow
committed