Eval bug: [Vulkan / MoltenVK] RDNA1 cards (5700XT, 5500XT) generating gibberish since b6264

[lksj92hs]

Name and Version

build: 6264 (043fb27) with Apple clang version 17.0.0 (clang-1700.0.13.5) for x86_64-apple-darwin24.5.0

Operating systems

Mac

GGML backends

Vulkan

Hardware

Core i7 10700K and Radeon Pro 5500 XT, Radeon 5700 XT, Radeon Pro VII
MoltenVK 1.4.0 official release

Models

unsloth_Qwen3-0.6B-GGUF_Qwen3-0.6B-Q4_K_M.gguf
unsloth_Qwen3-8B-GGUF_Qwen3-8B-Q4_K_M.gguf

Problem description & steps to reproduce

Since b6264, RDNA1 GPUs are generating gibberish (tested with the above Qwen3 models). Example command-line:

llama-server --host 0.0.0.0 -m ~/data/models/unsloth_Qwen3-0.6B-GGUF_Qwen3-0.6B-Q4_K_M.gguf -ngl 100 -dev Vulkan1

Note that the GCN card (Radeon Pro VII) works fine even with the latest releases.

Also note that the problem did not appear when testing under Windows 11 with GPU Radeon Pro 5500 XT, so it seems that it is MoltenVK specific.

First Bad Commit

I used git bisect to find the releases where the problem first appears and they are:

• b6257 (b1afcab): last OK release
• b6258 (611f419): output starts being strange but not total gibberish
• b6264 (043fb27): the output is total gibberish

I have run test-backend-ops on the relevant releases. The GPUs are:

• Radeon Pro VII - Vulkan0
• Radeon 5700XT - Vulkan1
• Radeon Pro 5500XT - Vulkan2

Output of test-backend-ops | grep FAIL:

• b6257 (b1afcab): last OK release backend_fails_6257.log
• b6258 (611f419): output starts being strange but not total gibberish backend_fails_6258.log
• b6264 (043fb27): the output is total gibberish backend_fails_6264.log
• b6387 has even more fails: backend_fails_6387.log

Relevant log output

llama-server --host 0.0.0.0 -m ~/data/models/unsloth_Qwen3-0.6B-GGUF_Qwen3-0.6B-Q4_K_M.gguf -ngl 100 -dev Vulkan1
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Vega (MoltenVK) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 5700 XT (MoltenVK) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 2 = AMD Radeon Pro 5500 XT (MoltenVK) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
build: 6264 (043fb27d) with Apple clang version 17.0.0 (clang-1700.0.13.5) for x86_64-apple-darwin24.5.0
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | OPENMP = 1 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 15
main: loading model
srv    load_model: loading model '/Users/ilko/data/models/unsloth_Qwen3-0.6B-GGUF_Qwen3-0.6B-Q4_K_M.gguf'
llama_model_load_from_file_impl: using device Vulkan1 (AMD Radeon RX 5700 XT) - 8176 MiB free
llama_model_loader: loaded meta data with 32 key-value pairs and 310 tensors from /Users/ilko/data/models/unsloth_Qwen3-0.6B-GGUF_Qwen3-0.6B-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3-0.6B
llama_model_loader: - kv   3:                           general.basename str              = Qwen3-0.6B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 0.6B
llama_model_loader: - kv   6:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   7:                          qwen3.block_count u32              = 28
llama_model_loader: - kv   8:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   9:                     qwen3.embedding_length u32              = 1024
llama_model_loader: - kv  10:                  qwen3.feed_forward_length u32              = 3072
llama_model_loader: - kv  11:                 qwen3.attention.head_count u32              = 16
llama_model_loader: - kv  12:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  13:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  14:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  16:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  17:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  18:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  19:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  20:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  21:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  22:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  23:            tokenizer.ggml.padding_token_id u32              = 151654
llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  26:               general.quantization_version u32              = 2
llama_model_loader: - kv  27:                          general.file_type u32              = 15
llama_model_loader: - kv  28:                      quantize.imatrix.file str              = Qwen3-0.6B-GGUF/imatrix_unsloth.dat
llama_model_loader: - kv  29:                   quantize.imatrix.dataset str              = unsloth_calibration_Qwen3-0.6B.txt
llama_model_loader: - kv  30:             quantize.imatrix.entries_count u32              = 196
llama_model_loader: - kv  31:              quantize.imatrix.chunks_count u32              = 688
llama_model_loader: - type  f32:  113 tensors
llama_model_loader: - type q4_K:  168 tensors
llama_model_loader: - type q6_K:   29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 372.65 MiB (5.24 BPW)
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3
print_info: vocab_only       = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 1024
print_info: n_layer          = 28
print_info: n_head           = 16
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 2
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 3072
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_finetuned   = unknown
print_info: model type       = 0.6B
print_info: model params     = 596.05 M
print_info: general.name     = Qwen3-0.6B
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 11 ','
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151654 '<|vision_pad|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 28 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   121.71 MiB
load_tensors:      Vulkan1 model buffer size =   372.65 MiB
.....................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     0.58 MiB
llama_kv_cache:    Vulkan1 KV buffer size =   448.00 MiB
llama_kv_cache: size =  448.00 MiB (  4096 cells,  28 layers,  1/1 seqs), K (f16):  224.00 MiB, V (f16):  224.00 MiB
llama_context:    Vulkan1 compute buffer size =   298.75 MiB
llama_context: Vulkan_Host compute buffer size =    14.01 MiB
llama_context: graph nodes  = 1098
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 4096
main: model loaded
main: chat template, chat_template: {%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0].role == 'system' %}
        {{- messages[0].content + '\n\n' }}
    {%- endif %}
    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0].role == 'system' %}
        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for forward_message in messages %}
    {%- set index = (messages|length - 1) - loop.index0 %}
    {%- set message = messages[index] %}
    {%- set current_content = message.content if message.content is defined and message.content is not none else '' %}
    {%- set tool_start = '<tool_response>' %}
    {%- set tool_start_length = tool_start|length %}
    {%- set start_of_message = current_content[:tool_start_length] %}
    {%- set tool_end = '</tool_response>' %}
    {%- set tool_end_length = tool_end|length %}
    {%- set start_pos = (current_content|length) - tool_end_length %}
    {%- if start_pos < 0 %}
        {%- set start_pos = 0 %}
    {%- endif %}
    {%- set end_of_message = current_content[start_pos:] %}
    {%- if ns.multi_step_tool and message.role == "user" and not(start_of_message == tool_start and end_of_message == tool_end) %}
        {%- set ns.multi_step_tool = false %}
        {%- set ns.last_query_index = index %}
    {%- endif %}
{%- endfor %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {%- set m_content = message.content if message.content is defined and message.content is not none else '' %}
        {%- set content = m_content %}
        {%- set reasoning_content = '' %}
        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
            {%- set reasoning_content = message.reasoning_content %}
        {%- else %}
            {%- if '</think>' in m_content %}
                {%- set content = (m_content.split('</think>')|last).lstrip('\n') %}
                {%- set reasoning_content = (m_content.split('</think>')|first).rstrip('\n') %}
                {%- set reasoning_content = (reasoning_content.split('<think>')|last).lstrip('\n') %}
            {%- endif %}
        {%- endif %}
        {%- if loop.index0 > ns.last_query_index %}
            {%- if loop.last or (not loop.last and (not reasoning_content.strip() == '')) %}
                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
            {%- else %}
                {{- '<|im_start|>' + message.role + '\n' + content }}
            {%- endif %}
        {%- else %}
            {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- endif %}
        {%- if message.tool_calls %}
            {%- for tool_call in message.tool_calls %}
                {%- if (loop.first and content) or (not loop.first) %}
                    {{- '\n' }}
                {%- endif %}
                {%- if tool_call.function %}
                    {%- set tool_call = tool_call.function %}
                {%- endif %}
                {{- '<tool_call>\n{"name": "' }}
                {{- tool_call.name }}
                {{- '", "arguments": ' }}
                {%- if tool_call.arguments is string %}
                    {{- tool_call.arguments }}
                {%- else %}
                    {{- tool_call.arguments | tojson }}
                {%- endif %}
                {{- '}\n</tool_call>' }}
            {%- endfor %}
        {%- endif %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking is false %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- endif %}
{%- endif %}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop

[jeffbolznv]

Can you try forcing off subgroup_add and subgroup_ballot? Maybe moltenvk has broken support for these.

[lksj92hs]

How to do that? Is there a runtime environment variable, or I need to recompile and if so, with what additional parameters for cmake?

[jeffbolznv]

There aren't precise enough env vars. You can just force them to false where they're set, at https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-vulkan/ggml-vulkan.cpp#L3650 and https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-vulkan/ggml-vulkan.cpp#L3643 (subgroup_arithmetic, not subgroup_add, I forgot the name).

[lksj92hs]

I made the changes in b6264, and yes, they solve the issue (it is subgroup_add in b6264, the name has changed after that to subgroup_arithmetic):

        device->subgroup_add = 0; //(vk11_props.subgroupSupportedStages & vk::ShaderStageFlagBits::eCompute) &&
                               //(vk11_props.subgroupSupportedOperations & vk::SubgroupFeatureFlagBits::eArithmetic);

        device->subgroup_shuffle = (vk11_props.subgroupSupportedStages & vk::ShaderStageFlagBits::eCompute) &&
                                   (vk11_props.subgroupSupportedOperations & vk::SubgroupFeatureFlagBits::eShuffle);

        device->subgroup_ballot = 0; // (vk11_props.subgroupSupportedStages & vk::ShaderStageFlagBits::eCompute) && (vk11_props.subgroupSupportedOperations & vk::SubgroupFeatureFlagBits::eBallot);

And with this change, there is no difference between the Radeon VII and the 5700XT GPUs:

sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[15,1023,1,1],scale=1.000000,max_bias=0.000000): FAIL
sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[1023,1023,1,1],scale=1.000000,max_bias=0.000000): FAIL
sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[15,1023,1,1],scale=0.100000,max_bias=0.000000): FAIL
sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[1023,1023,1,1],scale=0.100000,max_bias=0.000000): FAIL
  Backend Vulkan0: FAIL
sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[15,1023,1,1],scale=1.000000,max_bias=0.000000): FAIL
sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[1023,1023,1,1],scale=1.000000,max_bias=0.000000): FAIL
sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[15,1023,1,1],scale=0.100000,max_bias=0.000000): FAIL
sentinel mismatch: sent_3   SOFT_MAX_BACK(type=f32,ne=[1023,1023,1,1],scale=0.100000,max_bias=0.000000): FAIL
  Backend Vulkan1: FAIL

I don't have 6800 at hand (it won't fit in my eGPU case) but I'll try to test with it tomorrow to see if it has the issue. Still, I think it is better to disable this for all GPUs if Vulkan + MoltenVK is used.

EDIT: I just tested with the latest release b6402 and your patch is fixing the issue there as well. Strangely, 6402 seems to be 5-10% slower than 6264 but this is another matter.

[jeffbolznv]

Thanks for confirming. Do you want to make a PR, or shall I?

[jeffbolznv]

I've pushed a fix for the (unrelated) soft_max_back failures in #15861.

[lksj92hs]

It's better that you do it as I'm not familiar with the code at all. BTW, it turns out that only disabling subgroup_arithmetic is enough (without disabling subgroup_ballot) - all the fails are gone except the soft_max_back.

I guess this is the line which is responsible of the fact that GCN cards (like Radeon Pro VII) don't suffer from the issue:

llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp

Line 3021 in 3976dfb

const bool use_subgroups = device->subgroup_arithmetic && device->architecture != vk_device_architecture::AMD_GCN;

If you want, we can wait until I am able to test if RDNA2 (Radeon 6800) cards are affected to limit the scope of the change. Anything newer than RDNA2 is not supported at all on Macs, so we can safely ignore them.

[0cc4m]

If subgroup arithmetic is broken on MoltenVK, please also open an issue with them. Unless we misused the API (which is less likely since it works on other vendors, albeit not impossible), we can only work around this issue and the workaround should be removed again once it's fixed in MoltenVK. I've done that twice already with other bugs in the SPIRV to MSL compiler of MoltenVK, SPIRV-cross.

[lksj92hs]

Well, 6800 was mission impossible but I managed to fit a Radeon 6600 and it has the issue too. So both RDNA1 and RDNA2 have the issue (and the older cards are just not using subgroups).

If subgroup arithmetic is broken on MoltenVK, please also open an issue with them. Unless we misused the API (which is less likely since it works on other vendors, albeit not impossible), we can only work around this issue and the workaround should be removed again once it's fixed in MoltenVK. I've done that twice already with other bugs in the SPIRV to MSL compiler of MoltenVK, SPIRV-cross.

You are right but I don't think it would be fixed as Intel-based Macs with discrete GPUs are already dead and they are focused on Apple-silicone GPUs. Maybe we can make the workarounds affect only Apple discrete GPU code with some kind of build variable?

[0cc4m]

If subgroup arithmetic is broken on MoltenVK, please also open an issue with them. Unless we misused the API (which is less likely since it works on other vendors, albeit not impossible), we can only work around this issue and the workaround should be removed again once it's fixed in MoltenVK. I've done that twice already with other bugs in the SPIRV to MSL compiler of MoltenVK, SPIRV-cross.

You are right but I don't think it would be fixed as Intel-based Macs with discrete GPUs are already dead and they are focused on Apple-silicone GPUs. Maybe we can make the workarounds affect only Apple discrete GPU code with some kind of build variable?

Apple is focused on Apple-silicon, of course, but MoltenVK still supports older setups as far as I know. If it's a bug in the translation layer, they should be able to fix it. If it's a bug in the AMD driver on MacOS, then we probably can't do anything and have to work around it indefinitely.

[lksj92hs]

Yes, #15875 is definitely a duplicate of #15846