Using Batching? How does it work?

[zhzLuke96]

Hi all, I have some questions about batching in node-llama-cpp.

1. How does batching work?
   Is it processing tensors in parallel when using batching?

2. What is the relationship between sequences and batchSize in createContext()?

3. Does calling prompt automatically trigger batching?
   I couldn't find the source code for this, could someone explain how it works?

4. Does LlamaCompletion support batching?

5. Is there a way to directly control batching?
   Similar to the "RAW" mode inference mentioned here, is there a "RAW batching" feature?

Thanks for your help!

[giladgd]

1. Documentation: Using Batching.
2. Documentation: Object relationship
3. Documentation: Using Batching.
4. When using multiple context sequences from the same context in parallel, batching is automatically being used, regardless of how you use the context sequences (whether directly using .evaluate() or through a wrapper such as LlamaChatSession, LlamaCompletion, etc.).
5. You can customize the batching behavior by passing batching options when using .createContext(...); See BatchingOptions for more information on the batching options.
   You can also call .dispatchPendingBatch() on a context to manually dispatch a pending batch.

[zhzLuke96]

Thanks for the reply, I've read all these docs and roughly understand the batching capability provided by LlamaContext instance.

I still have some questions:

1. How can I estimate the VRAM usage for a batching process? Do you have any suggestions on this?
2. From my guessing, when generation processes exceed the sequences parameter, they automatically get pending to the next batching? Is this correct? The docs don't seem to mention what happens in these cases.
3. I think I can actually load multiple LLM models in VRAM by instantiating multiple LlamaModel objects. I'd like to know your thoughts on this kind of parallelization? When we have enough VRAM to support loading multiple models in memory, which approach would you recommend - single instance + batching or multiple instance parallelization?

Some background info:
I'm currently developing an open-source project called llm-boot which is built on node-llama-cpp. The goal is to provide a cross-platform solution that can convert *.gguf models into an OpenAI API compatible service.

[zhzLuke96]

and here’s some data from my testing:

• sequences: 1, prompt_array_length: 1: 2.07s
• sequences: 3, prompt_array_length: 3: 7.58s
• sequences: 1, prompt_array_length: 3: 4.79s

From this, it seems like batching actually makes the performance slower? I'm not sure what the issue is. Is this expected behavior?
ps: Batching should definitely be enabled, since more VRAM was used during inference.
pss: I tested using a temperature of 0, so there shouldn’t be any issues with randomness in inference.

[giladgd]

Using batching allows for more efficient evaluations that lead to more tokens being generated in the same time frame.
The efficiency gains depend on your hardware, but essentially it can slow down the generation per sequence but still allow generating more tokens in parallel from multiple sequences in the same time frame, thus you should measure the performance by the total tokens evaluated and generated under a context in a given time range.

Most of the memory is allocated during the creation of the context, so there's no additional runtime memory consumption that you should estimate, and batching has no impact on memory usage in this regard.

You can read more in the documentation on estimating model compatibility and resource consumption,
You can also access fileInsights on a loaded model to estimate resource consumption for a given configuration.

The number of sequences you set for a given context is fixed, and you can only use that number of sequences.
If you try to get more sequences than the ones allocated on the context (using .getSequence()) you'll get an error.
I'm working on making this dynamic so you won't have to specify the number of sequences you want in advance, but it's not ready yet.

There's no benefit to loading the same model more than once, as it doesn't hold any state.
It's more efficient (and thus usually performant) to create a single context with multiple sequences than to create multiple contexts with a single sequence.
The only benefit to creating multiple sequences is to have different options on different contexts (like flash attention or LoRA, for example).

If you mean to load and use different models simultaneously - you definitely can.
Just limit the context size of the contexts you create with them to ensure you have enough VRAM left for the other contexts you create (this will also be automatic soon, but it's not ready yet).