Without having looked at this PR, consider also fusing the Q, K, and V matrix multiplications into a single, batched operation. It's not going to reduce I/O but it's going to reduce kernel launch overhead and tail effects.

Note that with quantized types and GEGLU operator, I was getting NMSE as high as 1e-2

It looks like the newly added tests for f16 and f32 are failing on the CI for Tesla T4 GPU (cuda v 13) by quite a large amount, I notice that these might be the only tests with m=1 for mul_mat. On a rented T4 I don't see this problem (cuda v 12.6), so either it's a cuda version thing or an alignment thing

EDIT: it was neither, just a normal bug in not initing buf_iw_gate

This improved perf on ROCm backend as well. Gap to Vulkan is closing with this one!

PS D:\jam\llama.cpp\build\bin> .\llama-bench.exe -ngl 99 -mmp 0 -m .\llama-2-7b.Q4_0.gguf -fa 0,1 --device rocm0,vulkan0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 9070 XT, gfx1201 (0x1201), VMM: no, Wave Size: 32
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 9070 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat

Before PR:

After PR:

build: e6c48820 (6797)

BTW what else can we do to reduce this CPU overhead? I currently have a branch with a few other optimizations applied from other PR's https://github.com/ggml-org/llama.cpp/compare/master...jammm:llama.cpp:jam/gfx1201?expand=1 though this PR gives the most benefit.

@jammm depends on your model, kernel fusion helps more at the moment for MoE models. BTW sooner or later Vulkan will also adopt a similar thing so it's a shifting goalpost. For AMD specifically @JohannesGaessler recently made some changes in FA, be sure to keep include those as well. Another PR you can try related to CUDA graphs: #16548

@jammm depends on your model, kernel fusion helps more at the moment for MoE models. BTW sooner or later Vulkan will also adopt a similar thing so it's a shifting goalpost. For AMD specifically @JohannesGaessler recently made some changes in FA, be sure to keep include those as well. Another PR you can try related to CUDA graphs: #16548

Thanks! Will take a look at those. You're right re. the shifting goalposts.