Does LlamaChat handle truncating chat history?

[briancw]

From this page in the documentation I get the impression that chat history is automatically truncated to fit the contextSize when using LlamaChat or LlamaChatSession. However when I try to add more messages than fit in the current contextSize, it seems to delay indefinitely. I'm not sure if that's a bug or if the user is meant to handle truncating chat messages to not exceed the contextSize.

[giladgd]

Context shifting is handled automatically by node-llama-cpp so you don't have to handle it manually.
The reason you notice a delay in the middle of the response is because context shifting requires the re-evaluation of most of the new context history, and that can take some time, depending on the overall context size.

For example, if you have a context size of 100K tokens and all of it gets filled with chat history, node-llama-cpp will determine what are the parts of the chat history that are the most relevant (usually the most recent messages and the first system messages) and only render those it into tokens.
It attempts to free up 10% of the context size, so in this example, it'll attempt to fit as much of the chat history as possible in 90K tokens.
The newly compiled tokens then get evaluated to be loaded into the context state (there are some optimizations around that), and this process is as compute-intensive as evaluating an entire chat history on an empty context sequence.
In this example, 90K tokens will have to be evaluated to continue generating the current response, and it could take some time.
You can control the size of the context shift (by configuring the contextShift option on LlamaChatSession's .prompt() function) to attempt to incur fewer context shifts, but note that removing too much information from the current context state will affect the generated response quality.

I have tests for this implementation that seem to pass, but just to make sure, I've tested it again manually and it appears to work as expected.

[briancw]

Maybe there's an issue with small context sizes then. If you use a context of 1024 and try and pass in around 1200 tokens worth of context, it'll hang for, as I mentioned, an indefinite amount of time. I've waited as long as 30 minutes to see if it was just slow. I tested with different batch sizes to see if that was the issue but didn't have any luck there. Is there some reason for there to be a minimum context size for this to work?

[giladgd]

If you feed the model more tokens than the context size, it won't "see" all of these tokens since a context shift needs to happen on the chat history before the generation even starts, so in general, it's recommended to ensure the size of your system message + prompt + padding for the generation in tokens does not exceed the context size.

With that said, there currently seems to be an issue in such cases when under some conditions it'll trigger an infinite loop, which is the cause of what you experienced.
I found the cause for this and will release a fix soon.

Thanks for investigating it and sharing your findings :)

[giladgd]

I've included the fix for this issue in #375

[giladgd]

I've released a new version that includes a fix for this issue

[briancw]

It appears that the issue that I was running into was caused because I was testing using only "user" messages in the chat history which weren't being truncated. With 3.2.0, it throws an error about being unable to context shift rather than spinning indefinitely. I think the error message does a good job of conveying what the issue is and how to fix it. I'm now running into a different item with the size of the context used, but I'll make a new discussion for that to keep things organized.
Thank you for the fix and thanks for building such a killer library. node-llama-cpp is a pleasure to use and the API feels intuitive and well thought out.