Bug: llama.cpp with Vulkan not running on Snapdragon X + Windows (Copilot+PCs)

[AndreasKunar]

What happened?

The new Copilot+PCs with Qualcomm Snapdragon X processors (in my case a Surface 11 Pro with Snapdragon X Plus and 16GB RAM) are fast, and run llama.cpp on the CPU w/o issues. They also include a Vulkan driver and run the Vulkan samples w/o problems. But llama.cpp built with Vulkan does (now finally build,) but not run.

llama-cli is terminating on model-load with:
llama_model_load: error loading model: vk::Device::createComputePipeline: ErrorUnknown
llama_load_model_from_file: failed to load model
main: error: unable to load model

Name and Version

llama-cli version: 3378 (71c1121) with a quick-fix to compile (see #8446), built with MSVC 19.40.33812.0 for ARM64

built with:
Installed VulkanSDK for Windows x64, then built a Windows arm64 version of KhronosGroup/Vulkan-Loader vulkan-1.lib (+tested its functionality with tests+samples) and copied it to VulkanSDK lib-directory for llama.cpp building.

REM including Vulkan diagnostics
> cmake -B build -DGGML_VULKAN=1 -DGGML_VULKAN_DEBUG=1 -DGGML_VULKAN_MEMORY_DEBUG=1
> cmake --build build --config Release --target llama-cli

What operating system are you seeing the problem on?

Windows

Relevant log output

console output.txt
main.log
vulkaninfo.txt

[0cc4m]

I think that's the same bug that happens on Snapdragon phones: #5186

Some shader compiler bug in the Adreno driver. Might be good to report it to Qualcomm.

[AndreasKunar]

I think that's the same bug that happens on Snapdragon phones: #5186

Some shader compiler bug in the Adreno driver. Might be good to report it to Qualcomm.

@0cc4m thanks a lot. I will look into it.

I tried to run it also on WSL2 with its Microsoft CPU-emulated Vulkan-Driver. There it does not crash, but the results generated are garbage (as well as very slow).

I have tried to reach out to Qualcomm and will see if they answer.

[sykuang]

Have you experimented with the link https://apps.microsoft.com/detail/9nqpsl29bfff?hl=en-US&gl=US that facilitates the conversion of Vulkan shaders to D12 shaders?

[AndreasKunar]

Have you experimented with the link https://apps.microsoft.com/detail/9nqpsl29bfff?hl=en-US&gl=US that facilitates the conversion of Vulkan shaders to D12 shaders?

@skyan - thanks, its latest version installs automatically on the Surface with Snapdragon X. It implements a Microsoft Vulkan driver, which shows in addition to the Qualcomm Vulkan driver, and both show in vulkaninfo. llama.cpp's Vulkan backend picks the native Qualcomm driver (probably derived from their Android work), which seems to implement more/better Vulkan features than Microsoft's translation-driver to the (Qualcomm provided) native DX12 driver.

I will have a look, if I can test-tweak the llama.cpp Vulkan backend to use the Microsoft translation driver and report here, if I manage it and it works.

[0cc4m]

@AndreasKunar To try it you just have to manually pick the device, by setting the environment variable GGML_VK_VISIBLE_DEVICES to the index of the device you want on vulkaninfo --summary

[AndreasKunar]

@AndreasKunar To try it you just have to manually pick the device, by setting the environment variable GGML_VK_VISIBLE_DEVICES to the index of the device you want on vulkaninfo --summary

@0cc4m and @sykuang - thanks a lot!

Now with Microsoft's Vulkan to DX12 driver selected, the error changes from "llama_model_load: error loading model: vk::Device::createComputePipeline: ErrorUnknown" to "llama_model_load: error loading model: vk::Device::createComputePipeline: ErrorOutOfHostMemory". Trace with tinyllama-1.1b/ggml-model-f16.gguf below:

ggml_vk_instance_init()
ggml_vulkan: Found 1 Vulkan devices:
ggml_vk_print_gpu_info(1)
Vulkan0: Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU) (Dozen) | uma: 1 | fp16: 1 | warp size: 64
ggml_vk_get_device(0)
Initializing new vk_device
ggml_vk_find_queue_family_index()
ggml_vk_find_queue_family_index()
ggml_vk_create_queue()
ggml_vk_load_shaders(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU))
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_l, main, 3, 56, (128,128,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_m, main, 3, 56, (64,64,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_s, main, 3, 56, (32,32,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_aligned_l, main, 3, 56, (128,128,1), specialization_constants, 128)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_aligned_m, main, 3, 56, (64,64,1), specialization_constants, 64)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_aligned_s, main, 3, 56, (32,32,1), specialization_constants, 32)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_f16_l, main, 3, 56, (128,128,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_f16_m, main, 3, 56, (64,64,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_f16_s, main, 3, 56, (32,32,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_f16_aligned_l, main, 3, 56, (128,128,1), specialization_constants, 128)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_f16_aligned_m, main, 3, 56, (64,64,1), specialization_constants, 64)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f32_f16_aligned_s, main, 3, 56, (32,32,1), specialization_constants, 32)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_l, main, 3, 56, (128,128,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_m, main, 3, 56, (64,64,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_s, main, 3, 56, (32,32,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_aligned_l, main, 3, 56, (128,128,1), specialization_constants, 128)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_aligned_m, main, 3, 56, (64,64,1), specialization_constants, 64)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_aligned_s, main, 3, 56, (32,32,1), specialization_constants, 32)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_f32_l, main, 3, 56, (128,128,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_f32_m, main, 3, 56, (64,64,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_f32_s, main, 3, 56, (32,32,1), specialization_constants, 1)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_f32_aligned_l, main, 3, 56, (128,128,1), specialization_constants, 128)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_f32_aligned_m, main, 3, 56, (64,64,1), specialization_constants, 64)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_f16_f32_aligned_s, main, 3, 56, (32,32,1), specialization_constants, 32)
ggml_vk_create_pipeline(Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU), matmul_q4_0_f32_l, main, 3, 56, (128,128,1), specialization_constants, 128)
MESA: error: == VALIDATION ERROR =============================================
error: Total Thread Group Shared Memory storage is 33792, exceeded 32768.
Validation failed.

[0cc4m]

That means that the D3D12 translation layer doesn't provide enough shared memory for the big (_l) matrix multiplication shaders. You might be able to work around that by disabling them for Qualcomm GPUs (I don't think it's likely they are helpful here, they are meant for big dedicated GPUs) and not loading the pipelines in that case.

[sykuang]

@0cc4m and @AndreasKunar, I wanted to let you know that thanks to @0cc4m's input, I can run phi-3 on the Snapdragon X platform.

[AndreasKunar]

@sykuang - cool! What did you do exactly? Just reduce/edit the kernels (which?) and set GGML_VK_VISIBLE_DEVICES=1 (to the D2D12 driver)?

[sykuang]

@AndreasKunar, I've adjusted the code in ggml_vk_load_shaders by commenting out certain sections, which has resolved the issue where DirectX was reporting out-of-memory errors. You can refer to https://github.com/sykuang/llama.cpp/commit/b18e64826af69fe3765cd3b03c8dcd831a6bceca

[AndreasKunar]

@AndreasKunar, I've adjusted the code in ggml_vk_load_shaders by commenting out certain sections, which has resolved the issue where DirectX was reporting out-of-memory errors. You can refer to sykuang@b18e648

Thanks, I tried to get it to run, but once I offloaded Phi3-4k layers onto the GPU, either the results got strange or llama-cli crashed.

Vulkan performance on Snapdragon X Plus also was much worse than e.g. Q4_0_4_8 - Q4 vs. Q4_0_4_8 vs. Vulkan:

model	size	params	backend	threads	test	t/s
phi3 3B Q4_K - Medium	2.23 GiB	3.82 B	CPU	10	pp512	68.99 ± 11.74
phi3 3B Q4_K - Medium	2.23 GiB	3.82 B	CPU	10	tg128	26.42 ± 8.15
phi3 3B Q4_0_4_8	2.03 GiB	3.82 B	CPU	10	pp512	208.45 ± 73.38
phi3 3B Q4_0_4_8	2.03 GiB	3.82 B	CPU	10	tg128	34.83 ± 5.59

Vulkan0: Microsoft Direct3D12 (Snapdragon(R) X Plus - X1P64100 - Qualcomm(R) Adreno(TM) GPU) (Dozen) | uma: 1 | fp16: 1 | warp size: 64

model	size	params	backend	ngl	threads	test	t/s
phi3 3B Q4_0_4_8	2.03 GiB	3.82 B	Vulkan	99	10	pp512	40.67 ± 0.41
phi3 3B Q4_0_4_8	2.03 GiB	3.82 B	Vulkan	99	10	tg128	5.71 ± 0.20

build: 081fe43 (3441)

Can you maybe show your performance (llama-bench -p 512 -n 128) results? Currently for me it looks like Q4_0_4_8 quantization using the CPU's MATMUL is much faster than Vulkan.