Eval bug: Vulkan issues starting with 439342ea0be347ff279ec204719794df3b3108f6

[ddpasa]

Name and Version

Hello all, I've been having issues with Vulkan lately on my intel hardware (Iris Plus iGPU). I love Vulkan because it really speeds up image processing.

broken.txt
inxi.txt
vulkaninfo.txt
works.txt

I did a git bisect to determine the first broken commit and it is the following:

439342e is the first bad commit
commit 439342e (HEAD, tag: b7065)
Author: Jeff Bolz jbolz@nvidia.com
Date: Sat Nov 15 04:56:15 2025 -0600

vulkan: Use ggml_vk_tensor_subbuffer in mul_mat_vec(id) paths (#17244)

* vulkan: Use ggml_vk_tensor_subbuffer in mul_mat_vec(id) paths

* set allow_misalign

ggml/src/ggml-vulkan/ggml-vulkan.cpp | 631 +++++++++++++++++++++++++++++++++++----------------------------------------------------------------
1 file changed, 220 insertions(+), 411 deletions(-)

I am using the following command:

llama-mtmd-cli -ngl 0 -m /vlms/SmolVLM2-2.2B-Instruct-f16.gguf --mmproj /SmolVLM2-2.2B-Instruct-mmproj-f16.gguf --image ./tmp6zsf6lbv.jpg -p "Describe. It should be more than 10 words but less than 50 words."

I am also attaching a logs from a run that works, a run that doesn't work, inxi and vulkaninfo. Please let me know if you need additional information.

CPU version from the same commit works fine. Removing this commit also works fine.

Tagging @jeffbolznv for visibility

Operating systems

Linux

GGML backends

Vulkan

Hardware

Graphics:
Device-1: Intel Iris Plus Graphics G7 driver: i915 v: kernel
API: Vulkan v: 1.4.321 drivers: intel,llvmpipe surfaces: N/A

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Plus Graphics (ICL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
build: xxxx (xxxx) with cc (GCC) 15.2.1 20250813 for x86_64-pc-linux-gnu
llama_model_load_from_file_impl: using device Vulkan0 (Intel(R) Iris(R) Plus Graphics (ICL GT2)) (0000:00:02.0) - 10494 MiB free

Models

I tried with SmolVLM, but also seeing it with other models such as LFM2.

Problem description & steps to reproduce

llama-mtmd-cli -ngl 0 -m /vlms/SmolVLM2-2.2B-Instruct-f16.gguf --mmproj /SmolVLM2-2.2B-Instruct-mmproj-f16.gguf --image ./tmp6zsf6lbv.jpg -p "Describe. It should be more than 10 words but less than 50 words."

First Bad Commit

439342e

Relevant log output

warmup: flash attention is enabled
main: loading model: /vlms/SmolVLM2-2.2B-Instruct-f16.gguf
encoding image slice...
image slice encoded in 16314 ms
decoding image batch 1/1, n_tokens_batch = 81
Batch offset=0xc000 len=0x0 on queue 0 (aperture: 1218.7Mb, 0.0Mb VRAM only)
   BO: addr=0x000000058a200000-0x000000058b794fff size=  22100KB handle=00014 capture=0 vram_only=0 name=user
   BO: addr=0x0000000540000000-0x00000005524bffff size= 299776KB handle=00010 capture=0 vram_only=0 name=user
   BO: addr=0x0000000587800000-0x000000058a07ffff size=  41472KB handle=00013 capture=0 vram_only=0 name=user
   BO: addr=0x00000003c0000000-0x00000003c01fffff size=   2048KB handle=00002 capture=1 vram_only=0 name=dynamic pool
   BO: addr=0x0000000300000000-0x00000003001fffff size=   2048KB handle=00003 capture=1 vram_only=0 name=instruction pool
   BO: addr=0x00000000c0000000-0x00000000c01fffff size=   2048KB handle=00006 capture=1 vram_only=0 name=binding table pool
   BO: addr=0x0000000140000000-0x00000001401fffff size=   2048KB handle=00005 capture=1 vram_only=0 name=bindless surface state pool
   BO: addr=0x00000002c0000000-0x00000002c01fffff size=   2048KB handle=00007 capture=1 vram_only=0 name=indirect push descriptor pool
   BO: addr=0x0000000100000000-0x00000001001fffff size=   2048KB handle=00004 capture=1 vram_only=0 name=internal surface state pool
   BO: addr=0x0000000000200000-0x00000000003fffff size=   2048KB handle=00001 capture=1 vram_only=0 name=general pool
   BO: addr=0x0000000552600000-0x0000000553605fff size=  16408KB handle=00011 capture=0 vram_only=0 name=user
   BO: addr=0x0000000553800000-0x00000005877ddfff size= 851832KB handle=00012 capture=0 vram_only=0 name=user
   BO: addr=0xffffeffeffe00000-0xffffeffeffffffff size=   2048KB handle=00008 capture=1 vram_only=0 name=slab_parent
[New LWP 80882]
[New LWP 80881]
[New LWP 80880]
[New LWP 80879]
[New LWP 80878]
[New LWP 80876]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007fb610e9f042 in ?? () from /usr/lib/libc.so.6
#0  0x00007fb610e9f042 in ?? () from /usr/lib/libc.so.6
#1  0x00007fb610e931ac in ?? () from /usr/lib/libc.so.6
#2  0x00007fb610e931f4 in ?? () from /usr/lib/libc.so.6
#3  0x00007fb610f03dcf in wait4 () from /usr/lib/libc.so.6
#4  0x00007fb61456cb6b in ggml_print_backtrace () from /llama.cpp/build_vulkan/bin/libggml-base.so.0
#5  0x00007fb61457f379 in ggml_uncaught_exception() () from /llama.cpp/build_vulkan/bin/libggml-base.so.0
#6  0x00007fb6112b1eba in __cxxabiv1::__terminate (handler=<optimized out>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:48
warning: 48	/usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc: No such file or directory
#7  0x00007fb6112975d9 in std::terminate () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:58
58	in /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc
#8  0x00007fb6112b2176 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fb6144d7778 <typeinfo for vk::DeviceLostError>, dest=0x7fb61179cd80 <vk::DeviceLostError::~DeviceLostError()>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_throw.cc:98
warning: 98	/usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such file or directory
#9  0x00007fb61168639d in ggml_vk_submit(std::shared_ptr<vk_context_struct>&, vk::Fence) [clone .cold] () from /llama.cpp/build_vulkan/bin/libggml-vulkan.so.0
#10 0x00007fb61176d7a4 in ggml_vk_buffer_write_2d(std::shared_ptr<vk_buffer_struct>&, unsigned long, void const*, unsigned long, unsigned long, unsigned long) [clone .constprop.0] () from /llama.cpp/build_vulkan/bin/libggml-vulkan.so.0
#11 0x00007fb61176e406 in ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer*, ggml_tensor*, void const*, unsigned long, unsigned long) () from /llama.cpp/build_vulkan/bin/libggml-vulkan.so.0
#12 0x00007fb6145878b4 in ggml_backend_sched_graph_compute_async () from /llama.cpp/build_vulkan/bin/libggml-base.so.0
#13 0x00007fb61469afa0 in llama_context::graph_compute(ggml_cgraph*, bool) () from /llama.cpp/build_vulkan/bin/libllama.so.0
#14 0x00007fb61469ce63 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /llama.cpp/build_vulkan/bin/libllama.so.0
#15 0x00007fb6146a202f in llama_context::decode(llama_batch const&) () from /llama.cpp/build_vulkan/bin/libllama.so.0
#16 0x00007fb6146a2fce in llama_decode () from /llama.cpp/build_vulkan/bin/libllama.so.0
#17 0x00007fb614be6903 in mtmd_helper_decode_image_chunk () from /llama.cpp/build_vulkan/bin/libmtmd.so.0
#18 0x00007fb614be7c9d in mtmd_helper_eval_chunk_single () from /llama.cpp/build_vulkan/bin/libmtmd.so.0
#19 0x00007fb614be800c in mtmd_helper_eval_chunks () from /llama.cpp/build_vulkan/bin/libmtmd.so.0
#20 0x0000562078ae902c in eval_message(mtmd_cli_context&, common_chat_msg&) ()
#21 0x0000562078ae519a in main ()
[Inferior 1 (process 80874) detached]
terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Queue::submit: ErrorDeviceLost

[ddpasa]

Here is the git-bisect log:

$ git bisect log
# bad: [28175f857d21f7acb9bd060d61d8b073219fe59c] cmake : add option to build and link BoringSSL (#17205)
# good: [a864132ba546b6385922226480ee4e392aaa065c] devops: fix failing s390x docker build (#16918)
git bisect start '28175f857d21f7acb9bd060d61d8b073219fe59c' 'a864132ba546b6385922226480ee4e392aaa065c'
# good: [73460f62789a3ed02109d601a2653c3842f2074e] ggml-cpu: templateify ggml_compute_forward_rope_f32 and _f16 (#16805)
git bisect good 73460f62789a3ed02109d601a2653c3842f2074e
# bad: [22e1ce2f81c0495699502309dc279f1eac2dfdfe] webui: Fix clickability around chat processing statistics UI (#17278)
git bisect bad 22e1ce2f81c0495699502309dc279f1eac2dfdfe
# good: [0cfb19166b39a8c8ff2421bfcef7c3a595161367] metal: accelerated conv2d (#17175)
git bisect good 0cfb19166b39a8c8ff2421bfcef7c3a595161367
# good: [e1fcf8b09b8115156313c61bdd42186a7c7fb7be] model : add AfmoeForCausalLM support (#16477)
git bisect good e1fcf8b09b8115156313c61bdd42186a7c7fb7be
# bad: [c7b7db04456da9839ebd81ecde6f488c5fe41cbc] mtmd-cli: Avoid logging to stdout for model loading messages in mtmd-cli (#17277)
git bisect bad c7b7db04456da9839ebd81ecde6f488c5fe41cbc
# good: [234ae7d7bd0abd6814e83568553570225c516705] vulkan: skip all-negative-inf blocks in FA (#17186)
git bisect good 234ae7d7bd0abd6814e83568553570225c516705
# bad: [1568d13c2c6601fc902089ae53863bc0ef00259b] vulkan: implement ABS and NEG (#17245)
git bisect bad 1568d13c2c6601fc902089ae53863bc0ef00259b
# bad: [439342ea0be347ff279ec204719794df3b3108f6] vulkan: Use ggml_vk_tensor_subbuffer in mul_mat_vec(id) paths (#17244)
git bisect bad 439342ea0be347ff279ec204719794df3b3108f6
# first bad commit: [439342ea0be347ff279ec204719794df3b3108f6] vulkan: Use ggml_vk_tensor_subbuffer in mul_mat_vec(id) paths (#17244)

[ericcurtin]

This PR is open for another reason, but curious if you see the same on 26.04 packages @ddpasa ?

#17439

[jeffbolznv]

I haven't been able to reproduce this failure, and I also couldn't find any obvious bugs by inspection. I don't know how much mtmd behavior depends on the input image, but maybe it would help if you shared the image you used to reproduce this.

Does the failure still happen with top of tree master? We had a couple Intel-specific regressions recently.

[0cc4m]

I can't reproduce this and I don't see a problem in the commit. @ddpasa Please give https://github.com/ggml-org/llama.cpp/tree/0cc4m/vulkan-intel-mmv-fix a try, it's the only thing I noticed in the change. But I don't think this should make a difference.

If that doesn't help, please build with -DGGML_VULKAN_VALIDATE=ON and see if the validation layer show anything on your system. Also try upgrading your graphics driver, if it's not already up to date.

[ddpasa]

I ran with -DGGML_VULKAN_VALIDATE=ON and got the following, does this look reasonable? This is with the latest mainline using the mtmd binary and smolvlm.

Validation Warning: [ BestPractices-deprecated-extension ] | MessageID = 0xda8260ba
vkCreateInstance(): Attempting to enable deprecated extension VK_EXT_validation_features, but this extension has been deprecated by VK_EXT_layer_settings.

Validation Warning: [ BestPractices-specialuse-extension ] | MessageID = 0x675dc32e
vkCreateInstance(): Attempting to enable extension VK_EXT_validation_features, but this extension is intended to support use by applications when debugging and it is strongly recommended that it be otherwise avoided.

Validation Warning: [ BestPractices-deprecated-extension ] | MessageID = 0xda8260ba
vkCreateDevice(): Attempting to enable deprecated extension VK_EXT_pipeline_robustness, but this extension has been promoted to 1.4.0 (0x00404000).
Objects: 1
    [0] VkInstance 0x564752f63b20

Validation Warning: [ BestPractices-deprecated-extension ] | MessageID = 0xda8260ba
vkCreateDevice(): Attempting to enable deprecated extension VK_KHR_maintenance4, but this extension has been promoted to 1.3.0 (0x00403000).
Objects: 1
    [0] VkInstance 0x564752f63b20

Validation Warning: [ BestPractices-specialuse-extension ] | MessageID = 0x675dc32e
vkCreateDevice(): Attempting to enable extension VK_KHR_pipeline_executable_properties, but this extension is intended to support developer tools such as capture-replay libraries and it is strongly recommended that it be otherwise avoided.
Objects: 1
    [0] VkInstance 0x564752f63b20

Validation Warning: [ BestPractices-deprecated-extension ] | MessageID = 0xda8260ba
vkCreateDevice(): Attempting to enable deprecated extension VK_EXT_subgroup_size_control, but this extension has been promoted to 1.3.0 (0x00403000).
Objects: 1
    [0] VkInstance 0x564752f63b20

Validation Warning: [ BestPractices-deprecated-extension ] | MessageID = 0xda8260ba
vkCreateDevice(): Attempting to enable deprecated extension VK_KHR_16bit_storage, but this extension has been promoted to 1.1.0 (0x00401000).
Objects: 1
    [0] VkInstance 0x564752f63b20

Validation Warning: [ BestPractices-deprecated-extension ] | MessageID = 0xda8260ba
vkCreateDevice(): Attempting to enable deprecated extension VK_KHR_shader_non_semantic_info, but this extension has been promoted to 1.3.0 (0x00403000).
Objects: 1
    [0] VkInstance 0x564752f63b20

Validation Warning: [ BestPractices-deprecated-extension ] | MessageID = 0xda8260ba
vkCreateDevice(): Attempting to enable deprecated extension VK_KHR_shader_float16_int8, but this extension has been promoted to 1.2.0 (0x00402000).
Objects: 1
    [0] VkInstance 0x564752f63b20

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

Validation Warning: [ BestPractices-SpirvDeprecated_WorkgroupSize ] | MessageID = 0xd8a870c
(Warning - This VUID has now been reported 10 times, which is the duplicated_message_limit value, this will be the last time reporting it).
vkCreateComputePipelines(): pCreateInfos[0].stage is using the SPIR-V Workgroup built-in which SPIR-V 1.6 deprecated. When using VK_KHR_maintenance4 or Vulkan 1.3+, the new SPIR-V LocalSizeId execution mode should be used instead. This can be done by recompiling your shader and targeting Vulkan 1.3+.

One thing I'm now suspicious of now is that the behaviour seems to be non-deterministic. Sometimes I get the errors above, at other times it spews out nonsense but doesn't crash, or gets stuck indefinitely with high cpu usage. I will re-run the bisect with multiple runs this time.

[0cc4m]

Those warnings are expected.

[ddpasa]

indeed I will need to re-run the bisect. The behaviour is non-deterministic. I have confirmed that 554fd57 works fine on multiple different images (from October 15), and troubles start sometime after that.

@0cc4m are there any unit tests for the Vulkan backend that I can run? maybe those will be faster.

[0cc4m]

Yeah, you can run test-backend-ops, which tests each operator and compares it to the CPU result. However, it's not certain that it finds issues. We're dealing with a few issues recently that don't show up in the tests, only in real model use.

[ddpasa]

Yeah, you can run test-backend-ops, which tests each operator and compares it to the CPU result. However, it's not certain that it finds issues. We're dealing with a few issues recently that don't show up in the tests, only in real model use.

Ok, I might have something useful. I ran tests ( ctest -R test-backend-ops --progress -VV) for two commits: latest mainline b61de2b (broken) and 554fd57 (works).

31:   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=32,n=1024,k=16): �[1;32mOK�[0m
31:   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=2,n_used=2,b=0,m=32,n=8192,k=64): �[1;32mOK�[0m
31:   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=50,n=200,k=64): �[1;32mOK�[0m
31:   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=1,m=32,n=1024,k=16): �[1;32mOK�[0m
31:   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=2,n_used=2,b=1,m=32,n=8192,k=64): �[1;32mOK�[0m
31:   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=1,m=50,n=200,k=64): �[1;32mOK�[0m
31:   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=1,n_used=1,b=0,m=8,n=16,k=1): �[1;32mOK�[0m
31:   MUL_MAT_ID_FUSION(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=32,n=32,k=32,o=3,mul=0): Batch offset=0x4000 len=0x0 on queue 0 (aperture: 568.6Mb, 0.0Mb VRAM only)
31:    BO: addr=0x00000005555d0000-0x0000000555633fff size=    400KB handle=00015 capture=0 vram_only=0 name=user
31:    BO: addr=0x0000000557a00000-0x00000005601e6fff size= 139164KB handle=00010 capture=0 vram_only=0 name=user
31:    BO: addr=0x0000000540000000-0x00000005555cbfff size= 350000KB handle=00011 capture=0 vram_only=0 name=user
31:    BO: addr=0x00000003c0000000-0x00000003c01fffff size=   2048KB handle=00002 capture=1 vram_only=0 name=dynamic pool
31:    BO: addr=0x0000000300000000-0x00000003001fffff size=   2048KB handle=00003 capture=1 vram_only=0 name=instruction pool
31:    BO: addr=0x00000000c0000000-0x00000000c01fffff size=   2048KB handle=00006 capture=1 vram_only=0 name=binding table pool
31:    BO: addr=0x0000000140000000-0x00000001401fffff size=   2048KB handle=00005 capture=1 vram_only=0 name=bindless surface state pool
31:    BO: addr=0x00000002c0000000-0x00000002c01fffff size=   2048KB handle=00007 capture=1 vram_only=0 name=indirect push descriptor pool
31:    BO: addr=0x0000000100000000-0x00000001001fffff size=   2048KB handle=00004 capture=1 vram_only=0 name=internal surface state pool
31:    BO: addr=0x0000000000200000-0x00000000003fffff size=   2048KB handle=00001 capture=1 vram_only=0 name=general pool
31:    BO: addr=0x0000000300200000-0x00000003003fffff size=   2048KB handle=00013 capture=1 vram_only=0 name=instruction pool
31:    BO: addr=0x0000000300400000-0x00000003007fffff size=   4096KB handle=00016 capture=1 vram_only=0 name=instruction pool
31:    BO: addr=0x0000000300800000-0x0000000300ffffff size=   8192KB handle=00022 capture=1 vram_only=0 name=instruction pool
31:    BO: addr=0x0000000301000000-0x0000000301ffffff size=  16384KB handle=00023 capture=1 vram_only=0 name=instruction pool
31:    BO: addr=0x0000000302000000-0x0000000303ffffff size=  32768KB handle=00025 capture=1 vram_only=0 name=instruction pool
31:    BO: addr=0x0000000556c00000-0x000000055787bfff size=  12784KB handle=00026 capture=0 vram_only=0 name=user
31:    BO: addr=0xffffeffeffe00000-0xffffeffeffffffff size=   2048KB handle=00008 capture=1 vram_only=0 name=slab_parent
31: [New LWP 127014]
31: [New LWP 127013]
31: [New LWP 127012]
31: [New LWP 126380]
31: [New LWP 126379]
31: 
31: This GDB supports auto-downloading debuginfo from the following URLs:
31:   <https://debuginfod.archlinux.org>
31: Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
31: Debuginfod has been disabled.
31: To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
31: [Thread debugging using libthread_db enabled]
31: Using host libthread_db library "/usr/lib/libthread_db.so.1".
31: 0x00007fa8bbe9f042 in ?? () from /usr/lib/libc.so.6
31: #0  0x00007fa8bbe9f042 in ?? () from /usr/lib/libc.so.6
31: #1  0x00007fa8bbe931ac in ?? () from /usr/lib/libc.so.6
31: #2  0x00007fa8bbe931f4 in ?? () from /usr/lib/libc.so.6
31: #3  0x00007fa8bbf03dcf in wait4 () from /usr/lib/libc.so.6
31: #4  0x00007fa8bfa96b6b in ggml_print_backtrace () from /llama.cpp/build_vulkan/bin/libggml-base.so.0
31: #5  0x00007fa8bfaa9379 in ggml_uncaught_exception() () from /llama.cpp/build_vulkan/bin/libggml-base.so.0
31: #6  0x00007fa8bc2b1eba in __cxxabiv1::__terminate (handler=<optimized out>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:48
31: warning: 48	/usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc: No such file or directory
31: #7  0x00007fa8bc2975d9 in std::terminate () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:58
31: 58	in /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc
31: #8  0x00007fa8bc2b2176 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fa8bf5155b8 <typeinfo for vk::DeviceLostError>, dest=0x7fa8bc7a4840 <vk::DeviceLostError::~DeviceLostError()>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_throw.cc:98
31: warning: 98	/usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such file or directory
31: #9  0x00007fa8bc68739f in ggml_vk_submit(std::shared_ptr<vk_context_struct>&, vk::Fence) [clone .cold] () from /llama.cpp/build_vulkan/bin/libggml-vulkan.so.0
31: #10 0x00007fa8bc764c68 in ggml_vk_buffer_read(std::shared_ptr<vk_buffer_struct>&, unsigned long, void*, unsigned long) () from /llama.cpp/build_vulkan/bin/libggml-vulkan.so.0
31: #11 0x00007fa8bc765266 in ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer*, ggml_tensor const*, void*, unsigned long, unsigned long) () from /llama.cpp/build_vulkan/bin/libggml-vulkan.so.0
31: #12 0x000055669151dbff in tensor_to_float(ggml_tensor const*) ()
31: #13 0x000055669157abcf in test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*)::{lambda(int, ggml_tensor*, ggml_tensor*, void*)#1}::_FUN(int, ggml_tensor*, ggml_tensor*, void*) ()
31: #14 0x00007fa8bfab2ccc in ggml_backend_compare_graph_backend () from /llama.cpp/build_vulkan/bin/libggml-base.so.0
31: #15 0x000055669156a633 in test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*) ()
31: #16 0x00005566914fa5d7 in main ()
31: [Inferior 1 (process 126378) detached]
31: terminate called after throwing an instance of 'vk::DeviceLostError'
31:   what():  vk::Queue::submit: ErrorDeviceLost
1/1 Test #31: test-backend-ops .................Subprocess aborted***Exception: 292.46 sec

0% tests passed, 1 tests failed out of 1

The same test passes on 554fd57 but not on b61de2b.

[jeffbolznv]

Do you get any validation errors when running that test? You can run just that one test with test-backend-ops -o MUL_MAT_ID_FUSION(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=32,n=32,k=32,o=3,mul=0)

[ddpasa]

Do you get any validation errors when running that test? You can run just that one test with test-backend-ops -o MUL_MAT_ID_FUSION(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=32,n=32,k=32,o=3,mul=0)

Interestingly not:

$ test-backend-ops -o "MUL_MAT_ID_FUSION(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=32,n=32,k=32,o=3,mul=0)"
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Plus Graphics (ICL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
Testing 2 devices

Backend 1/2: Vulkan0
  Device description: Intel(R) Iris(R) Plus Graphics (ICL GT2)
  Device memory: 11660 MB (10211 MB free)

  MUL_MAT_ID_FUSION(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=32,n=32,k=32,o=3,mul=0): OK
  1/1 tests passed
  Backend Vulkan0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

Not sure what to make of this.

[jeffbolznv]

Hmm, it's frustrating that it doesn't happen when running the test alone. Is this devicelost reproducible with a full run? It may be an OOB memory access and could depend on memory allocations landing in a certain place.

FWIW, I don't think we actually do any fusion for this particular test case.