Without having looked at this PR, consider also fusing the Q, K, and V matrix multiplications into a single, batched operation. It's not going to reduce I/O but it's going to reduce kernel launch overhead and tail effects.

Note that with quantized types and GEGLU operator, I was getting NMSE as high as 1e-2

EDIT: this is fixed, it was a CUDA graphs bug. See comment below

This improved perf on ROCm backend as well. Gap to Vulkan is closing with this one!

PS D:\jam\llama.cpp\build\bin> .\llama-bench.exe -ngl 99 -mmp 0 -m .\llama-2-7b.Q4_0.gguf -fa 0,1 --device rocm0,vulkan0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 9070 XT, gfx1201 (0x1201), VMM: no, Wave Size: 32
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 9070 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat

Before PR:

After PR:

build: e6c48820 (6797)

BTW what else can we do to reduce this CPU overhead? I currently have a branch with a few other optimizations applied from other PR's https://github.com/ggml-org/llama.cpp/compare/master...jammm:llama.cpp:jam/gfx1201?expand=1 though this PR gives the most benefit.

@jammm depends on your model, kernel fusion helps more at the moment for MoE models. BTW sooner or later Vulkan will also adopt a similar thing so it's a shifting goalpost. For AMD specifically @JohannesGaessler recently made some changes in FA, be sure to keep include those as well. Another PR you can try related to CUDA graphs: #16548

@jammm depends on your model, kernel fusion helps more at the moment for MoE models. BTW sooner or later Vulkan will also adopt a similar thing so it's a shifting goalpost. For AMD specifically @JohannesGaessler recently made some changes in FA, be sure to keep include those as well. Another PR you can try related to CUDA graphs: #16548

Thanks! Will take a look at those. You're right re. the shifting goalposts.

There is something strange with running CUDA graphs with this, I get
Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGraphExecUpdate.

and the fusion doesn't go through, and/or there are failures with high NMSE. With GGML_DISABLE_CUDA_GRAPHS=1 everything seems to work fine. @slaren do you have ideas why this might be happening?

EDIT: The problem was CUDA graphs not re-capturing GGML_OP_GLU when the op_params have changed. This fixes it and also the high NMSE errors I was getting earlier

diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
index 1c283bc1..a0a94f62 100644
--- a/ggml/src/ggml-cuda/ggml-cuda.cu
+++ b/ggml/src/ggml-cuda/ggml-cuda.cu
@@ -2834,7 +2834,7 @@ static bool ggml_graph_node_has_matching_properties(ggml_tensor * node, ggml_gra
         }
     }

-    if (node->op == GGML_OP_SCALE &&
+    if ((node->op == GGML_OP_SCALE || node->op == GGML_OP_GLU) &&
         memcmp(graph_node_properties->op_params, node->op_params, GGML_MAX_OP_PARAMS) != 0) {
         return false;
     }

The problem was CUDA graphs not re-capturing GGML_OP_GLU when the op_params have changed.

When do the op_params of GGML_OP_GLU change?

When do the op_params of GGML_OP_GLU change?

I added both SWIGLU and GEGLU, the op param for GGML_OP_GLU changes in the test