CUDA: muh faster prompt processing for MoE models and small u-batch sizes#728

Merged

ikawrakow merged 23 commits intomainfrom

Aug 26, 2025

Commits on Aug 24, 2025

WIP: adding mainline mmq_id implementation
Iwan Kawrakow
committed
This seems to work
Iwan Kawrakow
committed
Now also -fmoe works
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed
WIP
Iwan Kawrakow
committed

Commits on Aug 25, 2025

WIP
Iwan Kawrakow
committed
This works for mainline supported quants
Iwan Kawrakow
committed
mmq_id: add iq2_k, iq2_k_r4
Iwan Kawrakow
committed
mmiq_id: don't assume row size is multiple of type size (per row scales)
Iwan Kawrakow
committed
mmiq_id: don't assume row size is multiple of type size
Iwan Kawrakow
committed
mmq_id: add iq2_ks
Iwan Kawrakow
committed
mmq_id: add iq2_kl
Iwan Kawrakow
committed
mmq_id: add iq3_ks
Iwan Kawrakow
committed
mmq_id: adding iq3_k, iq3_k_r4
Iwan Kawrakow
committed
mmq_id: add iq4_kss, iq4_ks, iq4_ks_r4
Iwan Kawrakow
committed
mmq_id: adding iq4_k, iq4_k_r4
Iwan Kawrakow
committed
mmq_id: adding iq5_ks, iq5_ks_r4
Iwan Kawrakow
committed
mmq_id: adding iq5_k, iq5_k_r4, q6_0
Iwan Kawrakow
committed
mmq_id: adding iq6_k
Iwan Kawrakow
committed
mmq_id: add iq1_s_r4
Iwan Kawrakow
committed
mmq_id: adding iq1_kt, iq2_kt
Iwan Kawrakow
committed
mmq_id: add iq3_kt, iq4_kt
Iwan Kawrakow
committed

Commits on Aug 26, 2025

Add CUDA fp8 header
Iwan Kawrakow
committed