Eval bug: Qwen3-30B-A3B-Q4_K_M: Vulkan ~10% slower than AVX2

[ZUIcat]

Name and Version

version: b5233 (ceda28e) with MSVC

Operating systems

Windows

GGML backends

Vulkan

Hardware

AMD 780m (8840u)

Models

Qwen3-30B-A3B-Q4_K_M.gguf
https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF/blob/main/Qwen_Qwen3-30B-A3B-Q4_K_M.gguf

Problem description & steps to reproduce

The AVX2 llama.cpp gives ~15 t/s, but Vulkan drops to ~9 t/s—unexpected slowdown.
Here are the command lines I used:
llama-server.exe -m "bartowski_Qwen3-30B-A3B-Q4_K_M.gguf" --host 0.0.0.0 --port 8090 --slots --props --metrics -np 1 -c 20480 -ngl 999 -ctk f16 -ctv f16 -fa --no-mmap --keep 0

First Bad Commit

No response

Relevant log output

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
build: 5233 (ceda28ef) with MSVC 19.43.34810.0 for x64
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8090, http threads: 15
main: loading model
srv    load_model: loading model 'bartowski_Qwen3-30B-A3B-Q4_K_M.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon 780M Graphics) - 16128 MiB free
llama_model_loader: loaded meta data with 41 key-value pairs and 579 tensors from bartowski_Qwen3-30B-A3B-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3 30B A3B
llama_model_loader: - kv   3:                           general.basename str              = Qwen3
llama_model_loader: - kv   4:                         general.size_label str              = 30B-A3B
llama_model_loader: - kv   5:                            general.license str              = apache-2.0
llama_model_loader: - kv   6:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv   7:                   general.base_model.count u32              = 1
llama_model_loader: - kv   8:                  general.base_model.0.name str              = Qwen3 30B A3B Base
llama_model_loader: - kv   9:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  10:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv  11:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv  12:                       qwen3moe.block_count u32              = 48
llama_model_loader: - kv  13:                    qwen3moe.context_length u32              = 32768
llama_model_loader: - kv  14:                  qwen3moe.embedding_length u32              = 2048
llama_model_loader: - kv  15:               qwen3moe.feed_forward_length u32              = 6144
llama_model_loader: - kv  16:              qwen3moe.attention.head_count u32              = 32
llama_model_loader: - kv  17:           qwen3moe.attention.head_count_kv u32              = 4
llama_model_loader: - kv  18:                    qwen3moe.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  19:  qwen3moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  20:                 qwen3moe.expert_used_count u32              = 8
llama_model_loader: - kv  21:              qwen3moe.attention.key_length u32              = 128
llama_model_loader: - kv  22:            qwen3moe.attention.value_length u32              = 128
llama_model_loader: - kv  23:                      qwen3moe.expert_count u32              = 128
llama_model_loader: - kv  24:        qwen3moe.expert_feed_forward_length u32              = 768
llama_model_loader: - kv  25:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  26:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  27:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  28:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  29:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  30:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  31:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  32:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  33:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - kv  36:                          general.file_type u32              = 15
llama_model_loader: - kv  37:                      quantize.imatrix.file str              = /models_out/Qwen3-30B-A3B-GGUF/Qwen_Q...
llama_model_loader: - kv  38:                   quantize.imatrix.dataset str              = /training_data/calibration_datav3.txt
llama_model_loader: - kv  39:             quantize.imatrix.entries_count i32              = 384
llama_model_loader: - kv  40:              quantize.imatrix.chunks_count i32              = 209
llama_model_loader: - type  f32:  241 tensors
llama_model_loader: - type q8_0:   48 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q5_K:   48 tensors
llama_model_loader: - type q6_K:   49 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 17.35 GiB (4.88 BPW)
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3moe
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 48
print_info: n_head           = 32
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_swa_pattern    = 1
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 6144
print_info: n_expert         = 128
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 30B.A3B
print_info: model params     = 30.53 B
print_info: general.name     = Qwen3 30B A3B
print_info: n_ff_exp         = 768
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors:      Vulkan0 model buffer size = 17596.42 MiB
load_tensors:          CPU model buffer size =   166.92 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 20480
llama_context: n_ctx_per_seq = 20480
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (20480) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     0.58 MiB
init: kv_size = 20480, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 48, can_shift = 1
init:    Vulkan0 KV buffer size =  1920.00 MiB
llama_context: KV self size  = 1920.00 MiB, K (f16):  960.00 MiB, V (f16):  960.00 MiB
llama_context:    Vulkan0 compute buffer size =   300.75 MiB
llama_context: Vulkan_Host compute buffer size =    80.01 MiB
llama_context: graph nodes  = 2935
llama_context: graph splits = 98
common_init_from_params: setting dry_penalty_last_n to ctx_size = 20480
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
common_chat_templates_init: failed to parse chat template (defaulting to chatml): Expected value expression at row 18, column 30:
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
                             ^
    {%- set index = (messages|length - 1) - loop.index0 %}

srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 20480
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8090 - starting the main loop
srv  update_slots: all slots are idle

[ggerganov]

Try to remove the -fa flag.

[ZUIcat]

Try to remove the -fa flag.

Surprisingly effective. Removing the -fa flag, Vulkan version reached ~25 t/s.

---

Here are my previous test results for others to test and reference:

• AV2 Version with -fa: ~15t/s
• Vulkan Version with -fa: ~9t/s
• Vulkan Version without -fa: ~25t/s

[0cc4m]

This is because Flash Attention on Vulkan is only supported on Nvidia GPUs with coopmat2 on an up-to-date driver. For any other device it will fall back to CPU, which is slow.

[ZUIcat]

This is because Flash Attention on Vulkan is only supported on Nvidia GPUs with coopmat2 on an up-to-date driver. For any other device it will fall back to CPU, which is slow.

Thank you. I hadn't noticed this before. I just tested with other models (e.g., gemma-2-27b-it-Q4_K_M) and confirmed that enabling -fa actually slows down performance when using Vulkan.

[jeffbolznv]

@ZUIcat please try flash attention with #13324. It may still be slower than -fa 0, but should be faster than CPU I hope.

[henfiber]

I noticed the same (and even worse) with my AMD 5600U, but without using Flash attention. Vulkan is 3-4x slower than the CPU in terms of PP, and about 10% slower in token generation.

• model: Unsloth/Qwen3-30B-A3B-GGUF:Q4_K_XL
• AMD 5600U (Zen3 APU 6c/12t, Radeon Vega 7, 2ch 64GB DDR4-3200)
• Fedora 41, Linux 6.12.13, Vulkan version: 1.4.309
• llama.cpp build: 141a908 (5298) (May 7, 2025)

Vulkan ./build/bin/llama-cli -m Qwen3-30B-A3B-Q4_K_XL -c 4096 -n 32768 -no-cnv -p '....'

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
...
llama_perf_context_print: prompt eval time =    5778,17 ms /   102 tokens (   56,65 ms per token,    17,65 tokens per second)
llama_perf_context_print:        eval time =   16153,25 ms /   226 runs   (   71,47 ms per token,    13,99 tokens per second)

CPU only ./build/bin/llama-cli -m Qwen3-30B-A3B-Q4_K_XL -c 4096 -n 32768 --device none -no-cnv -p '....'

llama_perf_context_print: prompt eval time =    1771,38 ms /   102 tokens (   17,37 ms per token,    57,58 tokens per second)
llama_perf_context_print:        eval time =   20916,04 ms /   336 runs   (   62,25 ms per token,    16,06 tokens per second)

I tested with different params (ngl, threads, fa, batch size) but the results didn't improve (-ngl 0 is even worse at PP of 10 t/s).

Note that, with dense models (llama3, qwen3-8b, qwen2.5-code-8b etc.), the PP performance I get on this APU is 1.8-2.5x better with Vulkan compared to the CPU (TG is the same).

Also for some reason, llama-bench crashes with this model although llama-server and llama-cli work properly (tested with up to 16k context).

The crash trace

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
~/.local/opt/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:5263: GGML_ASSERT(nei0 * nei1 <= 3072) failed
[New LWP 3696075]
[New LWP 3696074]
[New LWP 3696073]
[New LWP 3696072]
[New LWP 3696071]
[New LWP 3696064]
[New LWP 3696063]
This GDB supports auto-downloading debuginfo from the following URLs:

https://debuginfod.fedoraproject.org/

Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]

Debuginfod has been disabled.

To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

0x00007f26a9e04823 in wait4 () from /lib64/libc.so.6

#0  0x00007f26a9e04823 in wait4 () from /lib64/libc.so.6

#1  0x00007f26abdecd36 in ggml_abort () from ~/.local/opt/llama.cpp/build/bin/libggml-base.so

#2  0x00007f26aa4d421b in ggml_vk_mul_mat_id_q_f16(ggml_backend_vk_context*, std::shared_ptr<vk_context_struct>&, ggml_tensor const*, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, bool) () from ~/.local/opt/llama.cpp/build/bin/libggml-vulkan.so

#3  0x00007f26aa4ef0da in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_tensor*, int, ggml_tensor*, int, bool, bool, bool, bool) () from ~/.local/opt/llama.cpp/build/bin/libggml-vulkan.so

#4  0x00007f26aa4ef430 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) () from ~/.local/opt/llama.cpp/build/bin/libggml-vulkan.so

#5  0x00007f26abe02104 in ggml_backend_sched_graph_compute_async () from ~/.local/opt/llama.cpp/build/bin/libggml-base.so

#6  0x00007f26abfed751 in llama_context::graph_compute(ggml_cgraph*, bool) () from ~/.local/opt/llama.cpp/build/bin/libllama.so

#7  0x00007f26abff047e in llama_context::decode(llama_batch&) () from ~/.local/opt/llama.cpp/build/bin/libllama.so

#8  0x00007f26abff187b in llama_decode () from ~/.local/opt/llama.cpp/build/bin/libllama.so

#9  0x000000000040a11e in test_prompt(llama_context*, int, int, int) ()

#10 0x0000000000407157 in main ()

[Inferior 1 (process 3696062) detached]

^CAborted (core dumped)

EDIT: The crash issue has been resolved with latest build: 4696d56 (5384)

[ZUIcat]

@ZUIcat please try flash attention with #13324. It may still be slower than -fa 0, but should be faster than CPU I hope.

I just tested it. With -fa, it's about 22 t/s; without -fa, around 25 t/s.

version: build: 5335 (d891942) with MSVC 19.43.34810.0 for x64

[henfiber]

@ZUIcat Lower batch size (128 or 64) seems to increase PP with Vulkan on this model on Amd 5600U and Amd Strix Halo (8060S). Maybe you could test this with your 780M as well.

[lhl]

Just in case it's useful since @henfiber mentioned it, but here's some tests I've just run w/ Strix Halo (16c zen5 + 40cu gfx1151) in case it's useful:

All tests are run with llama-bench compiled from the latest 2025-05-15 HEAD (b5392) sequentially (-p 512, 1024, 4096) on unsloth's Qwen3-30B-A3B-UD-Q4_K_XL.gguf quant.

Backend	pp512	pp1024	pp4096
CPU -fa 0	250.77 ± 1.84	236.06 ± 1.13	181.83 ± 0.34
CPU -fa 1	246.54 ± 2.45	218.52 ± 0.83	122.17 ± 0.19
HIP -fa 0	405.14 ± 1.11	386.45 ± 1.08	322.03 ± 0.21
HIP -fa 1	366.03 ± 1.16	340.11 ± 0.95	237.20 ± 0.10
Vulkan -fa 0	69.78 ± 0.12	67.80 ± 0.25	66.43 ± 0.10
Vulkan -fa 1	70.94 ± 0.10	70.84 ± 0.05	69.51 ± 0.05
Vulkan -b 128 -fa 0	171.28 ± 1.00	171.02 ± 1.24	149.28 ± 0.33
Vulkan -b 128 -fa 1	171.62 ± 1.28	170.74 ± 0.91	160.60 ± 0.20

• In my tests -b 128 was the fastest pp512 for Vulkan (tried: default, 256, 128, 64).
• CPU uses up to PL power, HIP uses ~85W, Vulkan uses ~35W, Vulkan -b 128 gets to ~55W
• Vulkan fills GTT before spilling over to GART (HIP uses GTT-only)
• For gfx1151, Vulkan pp512 actually far exceeds CPU/HIP for the Llama 2 7B Q4_0 (standard test model) as well as for a Qwen3 32B Q8_0 dense model so the perf regression seems specific to the Qwen3 MoE models

This got me a bit curious and I ran a quick test on my workstation (EPYC 9274F + W7900) btw. Perf doesn't look quite so dire since Vulkan tends to be 2X slower than HIP anyway...

Backend	pp512
CPU -fa 0	174.07 ± 6.49
HIP -fa 0	851.69 ± 12.91
Vulkan -fa 0	270.98 ± 0.42
Vulkan -b 128 -fa 0	650.26 ± 5.78