Eval bug: Huge TG performance regression with b6969 (ROCm)

[8XXD8]

Name and Version

./llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 ROCm devices:
Device 0: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
Device 1: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
Device 2: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
version: 6969 (aa37417)
built with AOMP_STANDALONE_22.0_roc7-1 clang version 22.0.0_AOMP_STANDALONE_22.0_roc7-1 (https://github.com/ROCm/llvm-project 5e5ac6bb724fe52fe05a96cd4fee1aea0142d40c) for x86_64-unknown-linux-gnu

Operating systems

Linux

GGML backends

HIP

Hardware

Epyc 7B13 + 3X Radeon Pro VII

Models

Qwen3-VL-30B-A3B-Instruct Q8_0
Qwen3-VL-32B-Instruct Q8_0
Qwen3-VL-235B-A22B-Instruct Q8_0
GLM-4.6 Q5_K_M

Problem description & steps to reproduce

Token generation speed is much lower with b6969 while memory usage increased.
Every model is affected, to a different degree.
-fa on/off doesn't affect the result.

b6968 VRAM usage

============================================= ROCm System Management Interface =============================================
======================================================= Concise Info =======================================================
Device  Node  IDs              Temp    Power     Partitions          SCLK     MCLK     Fan     Perf    PwrCap  VRAM%  GPU%
              (DID,     GUID)  (Edge)  (Socket)  (Mem, Compute, ID)                                                         
============================================================================================================================
0       3     0x66a1,   28047  66.0°C  128.0W    N/A, N/A, 0         1654Mhz  1000Mhz  30.59%  manual  140.0W  80%    99%
1       1     0x66a1,   40382  54.0°C  132.0W    N/A, N/A, 0         1654Mhz  1000Mhz  30.59%  manual  140.0W  75%    99%
2       2     0x66a1,   52861  67.0°C  133.0W    N/A, N/A, 0         1654Mhz  1000Mhz  30.59%  manual  140.0W  77%    99%
============================================================================================================================
=================================================== End of ROCm SMI Log ====================================================

b6969 VRAM usage

============================================= ROCm System Management Interface =============================================
======================================================= Concise Info =======================================================
Device  Node  IDs              Temp    Power     Partitions          SCLK     MCLK     Fan     Perf    PwrCap  VRAM%  GPU%
              (DID,     GUID)  (Edge)  (Socket)  (Mem, Compute, ID)                                                         
============================================================================================================================
0       3     0x66a1,   28047  63.0°C  139.0W    N/A, N/A, 0         1654Mhz  1000Mhz  30.59%  manual  140.0W  91%    100%
1       1     0x66a1,   40382  52.0°C  121.0W    N/A, N/A, 0         1654Mhz  1000Mhz  30.59%  manual  140.0W  86%    100%
2       2     0x66a1,   52861  62.0°C  129.0W    N/A, N/A, 0         1654Mhz  1000Mhz  30.59%  manual  140.0W  88%    100%
============================================================================================================================
=================================================== End of ROCm SMI Log ====================================================

First Bad Commit

b6969

Relevant log output

b6968 llama-bench

llama-bench -m /home/user/text-generation-webui/models/Qwen3-vl-32b/Qwen3-VL-32B-Instruct-UD-Q8_K_XL.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 ROCm devices:
  Device 0: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 1: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 2: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |           pp512 |        110.06 ± 0.04 |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |           tg128 |         15.32 ± 0.01 |

build: 5b180c3d6 (6968)

llama-bench -m /home/user/text-generation-webui/models/Qwen3-vl-32b/Qwen3-VL-32B-Instruct-UD-Q8_K_XL.gguf -sm row
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 ROCm devices:
  Device 0: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 1: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 2: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |    sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | --------------: | -------------------: |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |   row |           pp512 |        186.17 ± 0.58 |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |   row |           tg128 |         22.19 ± 0.02 |

build: 5b180c3d6 (6968)

b6969 llama-bench

llama-bench -m /home/user/text-generation-webui/models/Qwen3-vl-32b/Qwen3-VL-32B-Instruct-UD-Q8_K_XL.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 ROCm devices:
  Device 0: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 1: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 2: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |           pp512 |        110.57 ± 0.05 |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |           tg128 |          8.36 ± 0.01 |

build: aa374175c (6969)

llama-bench -m /home/user/text-generation-webui/models/Qwen3-vl-32b/Qwen3-VL-32B-Instruct-UD-Q8_K_XL.gguf -sm row
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 ROCm devices:
  Device 0: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 1: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 2: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |    sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | --------------: | -------------------: |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |   row |           pp512 |        184.50 ± 1.41 |
| qwen3vl 32B Q8_0               |  36.76 GiB |    32.76 B | ROCm       |  99 |   row |           tg128 |          7.44 ± 0.00 |

build: aa374175c (6969)

[sgoll]

I can confirm this. With three MI300X and GLM 4.6 (GGUF, Q8), speed drops from ca. 35 tokens/sec to ca. 15 tokens/sec.

[eugr]

I can confirm this on Strix Halo (gfx1151, ROCm 7.10).

Before:

| model                          |       size |     params | backend    | ngl | n_ubatch | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | ---: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |          pp2048 |        916.16 ± 2.45 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |            tg32 |         53.22 ± 0.01 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |  pp2048 @ d4096 |        723.24 ± 0.49 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |    tg32 @ d4096 |         46.48 ± 0.10 |

After:

| model                          |       size |     params | backend    | ngl | n_ubatch | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | ---: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |          pp2048 |        919.15 ± 3.63 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |            tg32 |         44.15 ± 0.01 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |  pp2048 @ d4096 |        714.40 ± 6.69 |
| gpt-oss 120B MXFP4 MoE         |  59.02 GiB |   116.83 B | ROCm       |  99 |     2048 |  1 |    0 |    tg32 @ d4096 |         39.38 ± 0.05 |

git bisect results:

aa374175c30184aeb1813ec71fc68780dd073906 is the first bad commit
commit aa374175c30184aeb1813ec71fc68780dd073906 (tag: b6969)
Author: Johannes Gäßler <johannesg@5d6.de>
Date:   Thu Nov 6 14:05:47 2025 +0100

    CUDA: fix crash on uneven context without FA (#16988)

 ggml/src/ggml-cuda/ggml-cuda.cu | 12 ++++++------
 ggml/src/ggml-cuda/mmf.cu       | 12 +++++++++---
 ggml/src/ggml-cuda/mmf.cuh      |  2 +-
 ggml/src/ggml-cuda/mmvf.cu      |  8 +++++++-
 ggml/src/ggml-cuda/mmvf.cuh     |  2 +-
 src/llama-context.cpp           |  2 ++
 tests/test-backend-ops.cpp      | 44 ++++++++++++++++++--------------------------
 7 files changed, 44 insertions(+), 38 deletions(-)

[phoyd]

Can confirm aa37417 as culprit on 2xRadeon PRO VII, ROCm 7.0.2:

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | ROCm       |  99 |           pp512 |       1022.50 ± 6.82 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | ROCm       |  99 |           tg128 |         67.43 ± 0.07 |

build: aa374175c (6969)

Previous commit:

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | ROCm       |  99 |           pp512 |       1022.49 ± 6.85 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | ROCm       |  99 |           tg128 |        103.32 ± 0.66 |

build: 5b180c3d6 (6968)

[ggerganov]

The issue is that the context is no longer padded. It affects llama-bench results, but normal usage is OK. For now you can also use tg multiples of 256 until I provide a fix.

[8XXD8]

The issue is that the context is no longer padded. It affects llama-bench results, but normal usage is OK. For now you can also use tg multiples of 256 until I provide a fix.

I've had similar regression with llama-cli llama-server and mtmd-cli

I only included the llama-bench results because it's the most concise.

[ggerganov]

Can you demonstrate a llama-cli regression?

[8XXD8]

Can you demonstrate a llama-cli regression?

Yes, I will do that later today, I can't access the machine at work.

[phoyd]

Can you demonstrate a llama-cli regression?

$ ./bin/llama-cli -m ../models/lmstudio-community/gpt-oss-20b-GGUF/gpt-oss-20b-MXFP4.gguf -p "write a python3 program to approximate pi using three different methods" -no-cnv -n 1000

5b180c3

llama_perf_sampler_print:    sampling time =     133.31 ms /  1012 runs   (    0.13 ms per token,  7591.39 tokens per second)
llama_perf_context_print:        load time =    3957.46 ms
llama_perf_context_print: prompt eval time =     617.97 ms /    12 tokens (   51.50 ms per token,    19.42 tokens per second)
llama_perf_context_print:        eval time =    9343.15 ms /   999 runs   (    9.35 ms per token,   106.92 tokens per second)
llama_perf_context_print:       total time =   10595.15 ms /  1011 tokens
llama_perf_context_print:    graphs reused =        995
llama_memory_breakdown_print: | memory breakdown [MiB]   | total    free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - ROCm0 ((TM) Pro VII) | 16368 = 10266 + (5820 =  5613 +      56 +     150) +         281 |
llama_memory_breakdown_print: |   - ROCm1 ((TM) Pro VII) | 16368 = 10248 + (5852 =  5336 +      54 +     461) +         267 |
llama_memory_breakdown_print: |   - Host                 |                   629 =   586 +       0 +      42                |

aa37417

llama_perf_sampler_print:    sampling time =     142.55 ms /  1012 runs   (    0.14 ms per token,  7099.11 tokens per second)
llama_perf_context_print:        load time =    4344.51 ms
llama_perf_context_print: prompt eval time =     121.92 ms /    12 tokens (   10.16 ms per token,    98.42 tokens per second)
llama_perf_context_print:        eval time =   13726.13 ms /   999 runs   (   13.74 ms per token,    72.78 tokens per second)
llama_perf_context_print:       total time =   14474.51 ms /  1011 tokens
llama_perf_context_print:    graphs reused =        995
llama_memory_breakdown_print: | memory breakdown [MiB]   | total    free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - ROCm0 ((TM) Pro VII) | 16368 = 10268 + (5820 =  5613 +      56 +     150) +         279 |
llama_memory_breakdown_print: |   - ROCm1 ((TM) Pro VII) | 16368 = 10250 + (5852 =  5336 +      54 +     461) +         265 |
llama_memory_breakdown_print: |   - Host                 |                   629 =   586 +       0 +      42                |

107->73

Using -dev ROCm0 gives me a 126->80 drop.

[ggerganov]

Ah I see. I was thinking actually about #17037. I will reassign myself to that one.

Here I think there could be an additional issue originating from aa37417 - will let @JohannesGaessler take a deeper look.

To clarify, it's likely that both aa37417 and 85a7d86 have a compounding negative effect on the llama-bench numbers. But I believe that the llama-cli regression is only related to aa37417.