vulkan : add GGML_VK_FORCE_HEAP_INDEX env var

[gyf304]

Some vulkan devices (namely integrated graphics cards) have multiple memory heaps: a smaller dedicated memory and a larger shared memory.

ggml uses the first usable memory type, which usually resides on the smaller dedicated memory heap. This can likely cause allocation failures. This patch adds an environment variable that forces allocation on a specific memory heap.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[0cc4m]

Shouldn't this fallback already handle that situation? That's basically what it was introduced for. You have a smaller (DEVICE_LOCAL) and a larger heap (DEVICE_LOCAL, HOST_VISIBLE and HOST_COHERENT), so once the small one is exhausted you fall back to the larger one.
Can you add the vulkaninfo of the device where this fallback does not work and describe why?

[gyf304]

vulkaninfo.txt

The problem is that find_properties doesn't take in account about memory usage (as vk::PhysicalDeviceMemoryProperties reports total mem, not available mem)
On boot, with a few chrome tabs open, my laptop consumes 2.8/4GiB of the smaller heap, so if it tries to allocate 2GiB, it will fail.

One could argue the better fix is to either

• use VK_EXT_memory_budget for querying actual memory usage in find_properties, so it doesn't select a memory type that does not have enough available memory
• use a loop while allocating, if the allocation fails with ErrorOutOfDeviceMemory, try the next available memory type
• or a combination of both

I did the minimal viable solution here, which is to force allocation from a known bigger heap (in my case, heap 1).

[0cc4m]

@gyf304 Thank you.

Take a look at ggml-org/whisper.cpp#2451. This might be the kind of fallback that you described in your second point. Can you check whether that would fix your issue?

[ggerganov]

FYI, I just pulled ggml-org/whisper.cpp#2451 and will soon sync it upstream with ggml and llama.cpp. Not sure if it fixes the problem here, but it worked for a couple of whisper.cpp-related issues.