Misc. bug: Slower performance on newer models Apriel-1.5-15b, granite-4.0-h-tiny

[engrtipusultan]

Name and Version

bash  llama-server --version
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so
version: 6700 (3df2244)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

VULKANINFO

Vulkan Instance Version: 1.3.275

========
GPU0:
apiVersion = 1.4.318
driverVersion = 25.2.3
vendorID = 0x1002
deviceID = 0x15e7
deviceType = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
deviceName = AMD Radeon Graphics (RADV RENOIR)
driverID = DRIVER_ID_MESA_RADV
driverName = radv
driverInfo = Mesa 25.2.3 - kisak-mesa PPA
conformanceVersion = 1.4.0.0
deviceUUID = 00000000-0300-0000-0000-000000000000
driverUUID = 414d442d-4d45-5341-2d44-525600000000

hostnamectl
Operating System: Ubuntu 24.04.3 LTS
Kernel: Linux 6.14.0-33-generic
Architecture: x86-64
Hardware Vendor: GMKtec
Hardware Model: M5 PLUS
Firmware Version: M5 PLUS 1.03

Which llama.cpp modules do you know to be affected?

llama-server, llama-bench

Command line

Problem description & steps to reproduce

I have using Qwen3 A3B models (coder, instruct and thinking) and I am getting 12 tps token generation on Q8_0 quantization. I decided to give newer models a try which have A1B or A1.5B parameters. I though that token generation would higher on these models. But it is actually much less. I have given llama-bench commands that I run below. The same is observed in llama-server.

For example; Checking the Apriel-1.5-15b, I though it would be having double the token generation speed but it is horribly slow.
Also for some reason llama bench is detecting it as llama 34B Q8_0 and I am only getting 3.12 tps on generation as compared to 12.79 tps on qwen3 A3B. Prompt processing is also significantly slower.

Similarly I tried granite-4.0-h-tiny. It is not as slow as Apriel-1.5-15b but considering it is A1B and total 7B it is still has same token generation speed as Qwen3 A3B 30B. Granite tiny has 1B active parameters whereas Qwen3 A3B has three time more active parameters that is 3B.

I want to understand why it is so, is it correct this is how it is suppose to be or implementation of newer models will improve with time in the llama.cpp.

Excuse my ignorance if it something is obvious, please share why so.

First Bad Commit

I am not sure if it is actually a bug. I want to inquire.

Relevant log output

ash  llama-bench -m /home/tipu/AI/models/other/Qwen3-Coder-30B-A3B-Distill/Qwen3-30B-A3B-Instruct-Coder-480B-Distill-v2-Q8_0.gguf --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so

model	size	params	backend	ngl	threads	n_batch	n_ubatch	mmap	test	t/s
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	4	512	4096	0	pp512	94.84 ± 0.53
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	4	512	4096	0	tg128	12.79 ± 0.09

build: 3df2244 (6700)

 tipu-dev-machine   ~/Applications/llamaserver                                                                                    10:12:30 
 bash  llama-bench -m /home/tipu/AI/models/unsloth/Apriel/Apriel-1.5-15b-Thinker-Q8_0.gguf --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so

model	size	params	backend	ngl	threads	n_batch	n_ubatch	mmap	test	t/s
llama 34B Q8_0	14.28 GiB	14.43 B	Vulkan	99	4	512	4096	0	pp512	50.11 ± 0.15
llama 34B Q8_0	14.28 GiB	14.43 B	Vulkan	99	4	512	4096	0	tg128	3.12 ± 0.00

build: 3df2244 (6700)

 tipu-dev-machine   ~/Applications/llamaserver                                                                                    10:17:17 
 bash  llama-bench -m /home/tipu/AI/models/unsloth/Granite_4_tiny/granite-4.0-h-tiny-UD-Q8_K_XL.gguf --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 
load_backend: loaded RPC backend from /home/tipu/Applications/llamacpp/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /home/tipu/Applications/llamacpp/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tipu/Applications/llamacpp/libggml-cpu-haswell.so

model	size	params	backend	ngl	threads	n_batch	n_ubatch	mmap	test	t/s
granitehybrid ?B Q8_0	7.73 GiB	6.94 B	Vulkan	99	4	512	4096	0	pp512	210.72 ± 1.30
granitehybrid ?B Q8_0	7.73 GiB	6.94 B	Vulkan	99	4	512	4096	0	tg128	12.53 ± 0.02

[giuseppe]

I was looking into this issue and started working on: https://github.com/giuseppe/llama.cpp/tree/ssm-vulkan

Would it be possible for you to give it a try and see if it improves anything?

[engrtipusultan]

I was looking into this issue and started working on: https://github.com/giuseppe/llama.cpp/tree/ssm-vulkan

Would it be possible for you to give it a try and see if it improves anything?

Yes, sure. I cloned the repo and made vulkan build but I am getting error.

Edit: I think I did not checkout to the required branch and I am missing SDK. Let me check if I can do that.

tipu-dev-machine   ~/Development/GH/llama.cpp/build/bin  master ≡                                       15:01:03 
 bash  ./llama-bench -v -m /home/tipu/AI/models/unsloth/Apriel/Apriel-1.5-15b-Thinker-Q8_0.gguf 
| model                          |       size |     params | backend    |    threads |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ------------: | ---------------: |
llama_model_loader: loaded meta data with 42 key-value pairs and 435 tensors from /home/tipu/AI/models/unsloth/Apriel/Apriel-1.5-15b-Thinker-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Apriel-1.5-15B-Thinker
llama_model_loader: - kv   3:                           general.finetune str              = Thinker
llama_model_loader: - kv   4:                           general.basename str              = Apriel-1.5-15B-Thinker
llama_model_loader: - kv   5:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   6:                         general.size_label str              = 15B
llama_model_loader: - kv   7:                            general.license str              = mit
llama_model_loader: - kv   8:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   9:                   general.base_model.count u32              = 1
llama_model_loader: - kv  10:                  general.base_model.0.name str              = Apriel 1.5 15b Thinker
llama_model_loader: - kv  11:          general.base_model.0.organization str              = ServiceNow AI
llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/ServiceNow-AI/...
llama_model_loader: - kv  13:                               general.tags arr[str,2]       = ["unsloth", "text-generation"]
llama_model_loader: - kv  14:                          llama.block_count u32              = 48
llama_model_loader: - kv  15:                       llama.context_length u32              = 262400
llama_model_loader: - kv  16:                     llama.embedding_length u32              = 5120
llama_model_loader: - kv  17:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  18:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  19:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  20:                       llama.rope.freq_base f32              = 1000000000.000000
llama_model_loader: - kv  21:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  22:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  23:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  24:                           llama.vocab_size u32              = 131072
llama_model_loader: - kv  25:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = pixtral
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,131072]  = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,269443]  = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  33:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  34:            tokenizer.ggml.padding_token_id u32              = 11
llama_model_loader: - kv  35:                    tokenizer.chat_template str              = {# Unsloth template fixes #}\n{%- set ...
llama_model_loader: - kv  36:               general.quantization_version u32              = 2
llama_model_loader: - kv  37:                          general.file_type u32              = 7
llama_model_loader: - kv  38:                      quantize.imatrix.file str              = Apriel-1.5-15b-Thinker-GGUF/imatrix_u...
llama_model_loader: - kv  39:                   quantize.imatrix.dataset str              = unsloth_calibration_Apriel-1.5-15b-Th...
llama_model_loader: - kv  40:             quantize.imatrix.entries_count u32              = 336
llama_model_loader: - kv  41:              quantize.imatrix.chunks_count u32              = 136
llama_model_loader: - type  f32:   97 tensors
llama_model_loader: - type q8_0:  338 tensors
llama_model_load: error loading model: error loading model hyperparameters: invalid n_rot: 128, expected 160
llama_load_model_from_file: failed to load model
main: error: failed to load model '/home/tipu/AI/models/unsloth/Apriel/Apriel-1.5-15b-Thinker-Q8_0.gguf'

[giuseppe]

sorry, forgot to mention that I've tested only with granitehybrid. I take a look now with Apriel-1.5-15b-Thinker-Q8_0.gguf

[giuseppe]

that error happens before it even gets to the vulkan code, I am not able to reproduce with the model from https://huggingface.co/unsloth/Apriel-1.5-15b-Thinker-GGUF/tree/main:

Is it the same file?

$ sha256sum Apriel-1.5-15b-Thinker-Q8_0.gguf
bffa04967a9ebb47ebc946329a2f4ce9e2cbcddd4c5e59b862e221b36d1e71c2  Apriel-1.5-15b-Thinker-Q8_0.gguf

[engrtipusultan]

that error happens before it even gets to the vulkan code, I am not able to reproduce with the model from https://huggingface.co/unsloth/Apriel-1.5-15b-Thinker-GGUF/tree/main:

Is it the same file?

$ sha256sum Apriel-1.5-15b-Thinker-Q8_0.gguf
bffa04967a9ebb47ebc946329a2f4ce9e2cbcddd4c5e59b862e221b36d1e71c2  Apriel-1.5-15b-Thinker-Q8_0.gguf

Sorry I updated my reply earlier. Maybe you missed it.
I normally download the vulkan binary I have never built it myself. So I was having error because I did not have sdk.

What now I have done.
checked out to your branch after cloning repo.
installed the sdk. Release 1.4.313

Used following to build.

cmake -B build -DGGML_VULKAN=1
cmake --build build --config Release

After this I have run bench on granite. Same parameters as earlier. Token generation has improved:
(16.58−12.53)÷12.53×100 = 32.3 % Increase
and prompt processing has degraded to some extent. To be sure I ran it again.
(210−188)÷188×100 = 11 % decrease

 tipu-dev-machine   ~/Development/GH/llama.cpp  ssm-vulkan ≡                                                    15:31:03 
 bash  ./build/bin/llama-bench -m /home/tipu/AI/models/unsloth/Granite_4_tiny/granite-4.0-h-tiny-UD-Q8_K_XL.gguf --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

model	size	params	backend	ngl	threads	n_batch	n_ubatch	mmap	test	t/s
granitehybrid ?B Q8_0	7.73 GiB	6.94 B	Vulkan	99	4	512	4096	0	pp512	189.36 ± 0.71
granitehybrid ?B Q8_0	7.73 GiB	6.94 B	Vulkan	99	4	512	4096	0	tg128	16.35 ± 0.07

build: 4b21280 (6706)

 tipu-dev-machine   ~/Development/GH/llama.cpp  ssm-vulkan ≡                                                    15:33:13 
 bash  ./build/bin/llama-bench -m /home/tipu/AI/models/unsloth/Granite_4_tiny/granite-4.0-h-tiny-UD-Q8_K_XL.gguf --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 -r 8
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

model	size	params	backend	ngl	threads	n_batch	n_ubatch	mmap	test	t/s
granitehybrid ?B Q8_0	7.73 GiB	6.94 B	Vulkan	99	4	512	4096	0	pp512	187.99 ± 0.95
granitehybrid ?B Q8_0	7.73 GiB	6.94 B	Vulkan	99	4	512	4096	0	tg128	16.58 ± 0.04

build: 4b21280 (6706)

[engrtipusultan]

For Apriel-1.5-15b-Thinker

there is no increase in token generation but prompt processing is degrading.

bash  ./build/bin/llama-bench -m /home/tipu/AI/models/unsloth/Apriel/Apriel-1.5-15b-Thinker-Q8_0.gguf --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

model	size	params	backend	ngl	threads	n_batch	n_ubatch	mmap	test	t/s
llama 34B Q8_0	14.28 GiB	14.43 B	Vulkan	99	4	512	4096	0	pp512	47.15 ± 2.16
llama 34B Q8_0	14.28 GiB	14.43 B	Vulkan	99	4	512	4096	0	tg128	3.07 ± 0.02

For Old Qwen
benchmarks remain the same.

 tipu-dev-machine   ~/Development/GH/llama.cpp  ssm-vulkan ≡                                                    15:44:59 
 bash  ./build/bin/llama-bench -m /home/tipu/AI/models/other/Qwen3-Coder-30B-A3B-Distill/Qwen3-30B-A3B-Instruct-Coder-480B-Distill-v2-Q8_0.gguf --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

model	size	params	backend	ngl	threads	n_batch	n_ubatch	mmap	test	t/s
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	4	512	4096	0	pp512	94.53 ± 1.40
qwen3moe 30B.A3B Q8_0	30.25 GiB	30.53 B	Vulkan	99	4	512	4096	0	tg128	12.73 ± 0.05

[giuseppe]

I've tested on a NVIDIA L4, and this is what I get:

master (4e0388a)

# bin/llama-bench -r 5 -m ~/models/granite-4.0-h-tiny-UD-Q8_K_XL.gguf 

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     | 999 |           pp512 |       1419.59 ± 6.62 |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     | 999 |           tg128 |         33.57 ± 0.19 |


# bin/llama-bench -r 5  -m ~/models/Apriel-1.5-15b-Thinker-Q8_0.gguf

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 34B Q8_0                 |  14.28 GiB |    14.43 B | Vulkan     | 999 |           pp512 |       875.15 ± 11.83 |
| llama 34B Q8_0                 |  14.28 GiB |    14.43 B | Vulkan     | 999 |           tg128 |         16.46 ± 0.01 |

and the last version at https://github.com/giuseppe/llama.cpp/tree/ssm-vulkan (db8b8bc)

# bin/llama-bench -r 5 -m ~/models/granite-4.0-h-tiny-UD-Q8_K_XL.gguf

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     | 999 |           pp512 |      2114.02 ± 14.72 |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     | 999 |           tg128 |         80.59 ± 0.08 |


# bin/llama-bench -r 5 -m ~/models/Apriel-1.5-15b-Thinker-Q8_0.gguf

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 34B Q8_0                 |  14.28 GiB |    14.43 B | Vulkan     | 999 |           pp512 |       896.85 ± 10.46 |
| llama 34B Q8_0                 |  14.28 GiB |    14.43 B | Vulkan     | 999 |           tg128 |         16.49 ± 0.02 |

I've tested Apriel just for completeness, it should not be affected by my patches since it has no SSM, and the results are very volatile, I get very different results every time I run the test.

Using your same parameters, with granite I get:

master (4e0388a)

# bin/llama-bench --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 -r 8 -m ~/models/granite-4.0-h-tiny-UD-Q8_K_XL.gguf 
| model                          |       size |     params | backend    | ngl | threads | n_batch | n_ubatch | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------: | -------: | ---: | --------------: | -------------------: |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     |  99 |       4 |     512 |     4096 |    0 |           pp512 |       895.94 ± 18.80 |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     |  99 |       4 |     512 |     4096 |    0 |           tg128 |         32.08 ± 0.36 |

and (db8b8bcf77e154cc3b17d4bf0e2a2559af8a9409)

# bin/llama-bench --ubatch-size 4096 --batch-size 512 --threads 4 --mmap 0 -r 8 -m ~/models/granite-4.0-h-tiny-UD-Q8_K_XL.gguf 
| model                          |       size |     params | backend    | ngl | threads | n_batch | n_ubatch | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------: | -------: | ---: | --------------: | -------------------: |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     |  99 |       4 |     512 |     4096 |    0 |           pp512 |      2123.92 ± 27.01 |
| granitehybrid ?B Q8_0          |   7.73 GiB |     6.94 B | Vulkan     |  99 |       4 |     512 |     4096 |    0 |           tg128 |         80.44 ± 0.04 |

[giuseppe]

opened a PR: #16463

[engrtipusultan]

Huge benefits for NVIDIA vulkan. Hopefully drop in pp for amd vulkan igpu as mine is not due to this PR. But in any case tgps increases.

Thank you for your hard work.

Hopefully someone will look at Apriel-1.5-15b. It is very slow for your hardware as well. There is most probably something wrong there. I did not try granite small because low end hardware as mine, cannot do 9B active parameters decently but Apriel-1.5-15b is even more slow than dense GLM 9B on my machine.