Eval bug: model infer input "GGGGGGG"

[dhadialhda]

Name and Version

input error version: c247d06
input correct version: 15e0328

Operating systems

Windows

GGML backends

HIP

Hardware

AMD Radeon 880M

Models

phi-4-q4

Problem description & steps to reproduce

I identified a bug during testing on the AMD Radeon 880M. Bug description: When the length of the prompt exceeds the ubatch limit, the most recently updated code outputs "GGGGGG" during inference, as shown in Figure 1. I then attempted to roll back the code to the version from May, and the inference results were normal, as shown in Figure 2. I hope this issue can be resolved. Thank you.

First Bad Commit

I just test the 15e0328 ersion is correct.

Relevant log output

"GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG"

[IMbackK]

dupe of #15465, could you use git bisect between c247d06 and 15e0328 to find the acutal commit that introduced this issue for you?

[dhadialhda]

dupe of #15465, could you use git bisect between c247d06 and 15e0328 to find the acutal commit that introduced this issue for you?

Yes, I am narrowing down the scope.

[dhadialhda]

dupe of #15465, could you use git bisect between c247d06 and 15e0328 to find the acutal commit that introduced this issue for you?

I have located the root cause using binary search. Below is the log information. Could you please fix this bug as soon as possible?

C:\Users\yangl\Desktop\code\llama.cpp_new\llama.cpp>git bisect bad
eb39499 is the first bad commit
commit eb39499
Author: Shawn yang 137684499+Yangxiaoz@users.noreply.github.com
Date: Sat May 31 14:48:04 2025 +0800

CUDA: add a prop in ggml_cuda_device_infor for distinguish iGPU or dGPU in cuda (#13856) (#13895)

* 1.  add "integrated" in ggml_cuda_device_info for distinguish whether it is Intergrate_gpu or discrete_gpu
2. Adjust the func:"ggml_backend_cuda_device_supports_buft" for this new feature

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Adjusted code indentation

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Fixed incorrect setting of variable types

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Adjusted the judgment logic

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* add a host_buft assert in case of integrated_cuda_device with func:'evaluate_and_capture_cuda_graph()'

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Add a defensive security assert

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Adjusted the support judgment logic.

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* revoke the suggest commit changes due to it's not applicable in jetson_device

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Add parentheses to enforce operator precedence​

Co-authored-by: Diego Devesa <slarengh@gmail.com>

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Fix ci bug: add a spaces

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: yangxiao <yang_xl@tju.edu.cn>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: yangxiao <yangxl_zz@qq.com>
Co-authored-by: Diego Devesa <slarengh@gmail.com>

ggml/src/ggml-cuda/common.cuh | 1 +
ggml/src/ggml-cuda/ggml-cuda.cu | 20 ++++++++++++++------
2 files changed, 15 insertions(+), 6 deletions(-)

C:\Users\yangl\Desktop\code\llama.cpp_new\llama.cpp>git bisect reset
Previous HEAD position was eb39499 CUDA: add a prop in ggml_cuda_device_infor for distinguish iGPU or dGPU in cuda (#13856) (#13895)
HEAD is now at e562eec CUDA: fix typo in FlashAttention code (#13926)

[IMbackK]

So im not actually sure how this part of ggml works, i presume *_supports_buft tells ggml_backend that the device can map host buffers with good performance. On cuda/hip *HostRegister has to be used to make this work and ggml_backend_cuda_register_host_buffer is intended to do so (but only when GGML_CUDA_REGISTER_HOST is set and not on hip for some reason, seams like an oversight) and i have no idea how ggml_backend_cuda_register_host_buffer ends up getting called anyhow.

@slaren could you enlighten us?

[IMbackK]

anyhow: #15615

[slaren]

The supports_buft function tells the scheduler the buffer types that the backend can use. The graph_compute function may receive a graph with tensors allocated in these buffer types. ggml_backend_cuda_register_host_buffer was intended to be used to pin memory mapped buffers, it's not used at the moment AFAIK.

[IMbackK]

ok, but then if ggml_backend_cuda_register_host_buffer is not used, then allowing ggml_backend_buft_is_cuda_host(buft) in supports_buft means that ggml will use ggml_cuda_host_malloc in place of ggml_cuda_device_malloc for some tensors, based on some logic correct?

Over all this also seams like a parallel mechanism to ggml_cuda_device_malloc using cudaMallocManaged when GGML_CUDA_ENABLE_UNIFIED_MEMORY is enabled, which will also use host memory/gtt on igpus.

[slaren]

It means that the CUDA backend may receive tensors allocated in the host buffer type. The host buffer type is used in place of the CPU buffer type when available, and when mmap is disabled.

[IMbackK]

@dhadialhda just a heads up: I tried repouceing this by forcing this path on a dgpu and everything seamed to work OK, I think your issue might be more indicative of there being a bug in rocr rather than llamacpp, but I'm not quite sure yet. Meanwhile it would help if you could give us the rocm version(s) you see this issue on.

I will be traveling for 2 weeks and will get back to this after.