feat: Vulkan split allocations

[raftario]

Feature Description

Vulkan buffer allocations cap out at 4GB, which isn't enough for some models. This was recently addressed upstream.

The Solution

It would be nice to get a new release on npm with those changes.

Considered Alternatives

Building locally works in the meantime. I might also add ROCm support if I find the time.

Additional Context

No response

Related Features to This Feature Request

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

N/A

[raftario]

Realised this is already in the release and I was actually running into a separate issue where the context buffer was too large.

[giladgd]

@raftario node-llama-cpp automatically attempts to retry creating a context with an incrementally smaller size when the allocation fails (usually under Vulkan with huge sizes like what happened to you) if you don't set an explicit context size, so I recommend configuring contextSize with min and max values to define the range of sizes that are acceptable for your needs instead of setting it to a specific number.

Regarding ROCm, I'd love if you can add support for it. I haven't done it myself since I don't have access to a machine with an AMD GPU to develop this.
If you get to it, check out my comment #479 (comment) for guidance on what's needed to be done to implement it, and don't hesitate to ask for help if you get stuck on something.

[raftario]

@giladgd FWIW it might be nice to have the option to apply that retry logic to the batch size rather than context size for people who might wanna take advantage of the largest context size possible.

[giladgd]

@raftario Have you encountered issues with big batch sizes on Vulkan? What batch size caused issues for you?

There's not much advantage to increasing the batch size by much, the default is very optimal for most use cases.
Moreover, increasing the batch size too much can waste more memory and decrease performance due to increased overhead.
You won't notice any difference in token generation performance with a bigger batch size, and would probably notice a negligible boost in prompt processing time (only of huge prompts), but it's not worth the performance hit in token generation.

The main kind of scenarios I think would benefit the most from a huge batch size is when you use a small model (e.g. a model with less than 4B activations) on an extremely powerful GPU (like a H200) and process thousands of context sequences at once, but if that's the case you'd probably want to use CUDA and not Vulkan anyway to utilize every bit of optimization that this GPU is capable of.

[raftario]

@giladgd I meant the opposite, that instead of decreasing the context size until the allocation succeeds it would be useful to have the option to decrease the batch size. In this case I was using Mistral Small, which has a 128k context, and wanted to make use of all that context. But with a batch size of 512 this results in a 8GB compute buffer allocation and Vulkan throws.

I ended up manually specifying a batch size of 128, but what I would like is the option for the library to automatically retry with smaller and smaller batch sizes for me, since like you said 512 is at the upper end of useful batch sizes for most use cases.

[giladgd]

@raftario I wasn't aware that using a smaller batch size actually helps allocating a larger context with Vulkan. Thank you for sharing that!
I'll investigate it further to see whether this can be efficiently automated and what would be a good default balance.

[raftario]

Thank you for looking into this, and also for the amazing bindings !