Misc. bug: llama-server: Invalid Reply Generated with Vulkan Backend with VK_KHR_cooperative_matrix Support

[majunlichn]

Name and Version

llama-server version: 4292 (1a05004)

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4080 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | matrix cores: KHR_coopmat
version: 4292 (1a050047)
built with MSVC 19.42.34435.0 for x64

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Problem description & steps to reproduce

Build:

cmake -S . -B build -DGGML_VULKAN=ON -DGGML_OPENMP=OFF -DGGML_STATIC=OFF -DBUILD_SHARED_LIBS=ON
cmake --build build --config Release

Command:

llama-server.exe -m Phi-3-mini-4k-instruct-fp16.gguf -ngl 100

The model is downloaded from: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/blob/main/Phi-3-mini-4k-instruct-fp16.gguf (other models may also reproduce the issue).

Open browser: http://127.0.0.1:8080 and start a new conversation with "Hi" (tested on RTX 4080 with driver 565.90):

I got invalid reply with code "#" repeatedly.
runlog: llama-server-runlog-verbose.txt

If set $env:GGML_VK_DISABLE_COOPMAT=1 and $env:GGML_VK_DISABLE_COOPMAT2=1, I can get normal reply "Hello! How can I assist you today?".

First Bad Commit

I think the issue is related to: #10597
Commit: 3df784b

Relevant log output

See https://github.com/user-attachments/files/18058540/llama-server-runlog-verbose.txt.

[0cc4m]

Thank you for the report. I don't have a Windows system with an Nvidia GPU to test this, but I'll check if I can reproduce it on Linux.

@jeffbolznv You tested the coopmat1 branch on Windows on an Ada GPU, right?

[lenhone]

The issue also appears to affect llama-cli on Arch Linux with an AMD 7900GRE where the output is repeated tab white-space (\x09).

./llama-cli -cnv -m ./EXAONE-3.5-7.8B-Instruct-Q8_0.gguf -p "You are EXAONE model from LG AI Research, a helpful assistant." -ngl 99

> hello
													

When GGML_VK_DISABLE_COOPMAT=1 is used the response is normal.

[0cc4m]

@majunlichn @lenhone Can you please also run ./test-backend-ops -o MUL_MAT (Linux) / test-backend-ops.exe -o MUL_MAT (Windows) and post the output?

[jeffbolznv]

I can't reproduce it on RTX 4070 with the same driver.

[majunlichn]

test-backend-ops-win-RTX4080-MUL_MAT.txt

BTW, my colleague also tested AMD's RX 7900 (Navi31) on Linux with RADV driver, but also got invalid replies ("#" repeatedly).

@jeffbolznv Does coopmat support requires any special build option or parameter to run?

[jeffbolznv]

No special build options or parameters are necessary. What driver version are you using? If it's not the latest, can you try updating?

[majunlichn]

RTX4080 fails with latest 566.36 GRD driver or 566.14 Studio driver on Windows (test-backend-ops.exe -o MUL_MAT).

[majunlichn]

With my colleague's help I confirmed that the issue is related to VulkanSDK version - 1.3.268.0 is good while 1.3.296.0 fails.
So the issue is related to glslc/glslang (the compilation of GLSL=>SPIRV).

[GinShio]

Passes if glslc optimization is disabled.

diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp b/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp
index bc6fca50..56b7a5d4 100644
--- a/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp
+++ b/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp
@@ -209,5 +209,5 @@ void string_to_spv_func(const std::string& _name, const std::string& in_fname, c
     #ifdef _WIN32
-        std::vector<std::string> cmd = {GLSLC, "-fshader-stage=compute", target_env, "-O", "\"" + in_path + "\"", "-o", "\"" + out_fname + "\""};
+        std::vector<std::string> cmd = {GLSLC, "-fshader-stage=compute", target_env, "-O0", "\"" + in_path + "\"", "-o", "\"" + out_fname + "\""};
     #else
-        std::vector<std::string> cmd = {GLSLC, "-fshader-stage=compute", target_env, "-O", in_path, "-o",  out_fname};
+        std::vector<std::string> cmd = {GLSLC, "-fshader-stage=compute", target_env, "-O0", in_path, "-o",  out_fname};
     #endif

Environment:

• glslc: 2024.3
• vulkan: radv 03fc4838130
• GPU: Navi33

[jeffbolznv]

Thanks for narrowing this down! I can reproduce it locally now, and I'll make a PR to disable spirv-opt for these shaders.

The bug must be fixed in top of tree spirv-tools, since I didn't reproduce it with a more recent build. I had fixed a few spirv-opt issues in KhronosGroup/SPIRV-Tools#5865, probably one of those.