Custom Context Shift Strategy implementation

[physimo]

Hi, it's me again (I'm sorry if I asked too many questions...)
I want to know, how do you actually implement custom context shift strategy?
The idea on #359 answer, which talks about making the model summarize the conversation so far and put it into the system prompt as key points of the conversation is actually interesting and I want to try implementing it.
But since I actually had no experience on making a custom context shift strategy, I tried to fiddle around in the dark.

Here's what I've tried:

• Limitting the context size to 1024 * 0.5, which results in 512 tokens
• Observing lastContextWindow and lastContextShiftMetadata to know how does the default strategy shift the context, and the result of the shifting
• Talking to the model to try triggering context shifting, while keep doing point 2 (observing)

The reason I limitted the context size to 512 is to speed up the process so that the context shift happens earlier in the conversation. The conversation itself is short exchanges and pleasantries, but when context shift happens, it always results in this error:

Error: The default context shift strategy did not return a history that fits the context size. This may happen due to the system prompt being too long

I've tried several times by increasing the context size, but still the context shift results in this error.
So I'll just come here and ask things:

1. What's the purpose of the lastContextShiftMetadata?
2. Is it correct to assume that history got nothing to do with the context shifting? If so, what's the point of lastContextWindow object, since history also provides the whole conversation context?
3. I see that the custom shift strategy also accepts Promise result, and from what I see, unless you store the response buffer in an array, the generateResponse() method will not return the consolidated result of the generation unless context shifting is done. If I'm using model to summarize the whole conversation into key points, won't that take a long time before the method will actually return the response?
4. If 3 is true (that it will take a while for the method to return response until context shifting is done), is it possible to manually shift the context so that it happens outside of active time?
5. Also, is it correct to assume that when context shift happens, the only affected object is lastContextWindow, and history will remain as it is?

If it matters, I'm using LlamaChat, not LlamaChatSession since I need to implement node-llama-cpp with other library.
Again, I'm sorry for asking too many question here.
Please bear with me.

[giladgd]

The purpose of the context shift is to free space from the context sequence to make room for the next generation, so it only happens when the context sequence is full.
LlamaChat and LlamaChatSession attempt to use the existing context state as much as possible to avoid redundant context shifts, which is why you have to pass lastContextWindow.

The reason you pass lastContextShiftMetadata is to make it easier for the context shift algorithm to do the next context shift.
Every time the context shift function is called, it gets the full chat history, and the number of tokens the "rendered" chat state needs to fit - you can use the chat wrapper's generateContextState function to get the "rendered" context state of the given history and then tokenize it using the given tokenizer to check whether its length fits within the constraint of the maxTokensCount value.
That function needs to return a compressed chat history that when "rendered" and tokenized will be under the maxTokensCount value, otherwise you get the error you mentioned.
That function can also return a metadata value that will be passed back to it the next time a context shift happens, which can be used to optimize the next context shift and avoid redundant computations.

You can use another context sequence to summarize messages in the current chat for the context shift.
When a context shift happens, you'll see that the context window of the chat has changed.

I see that I haven't added a section for implementing a custom context shift function, so I'll try to get to it soon to make it easier to use.

Let me know whether this helped you.

[physimo]

I see that I haven't added a section for implementing a custom context shift function, so I'll try to get to it soon to make it easier to use.

That's great to know!

Though I still have 2 questions, just to make sure that I get this correctly:
The chat's context is stored in lastContextWindow, and when context shift happens, the chat history (LlamaChatResponse.lastEvaluation.cleanHistory) is left untouched right?
Also,
The only object that gets compressed (shortened, or summarized to be below the maxTokensCount value) when context shift happens, is the lastContextWindow, correct?

[scenaristeur]

Hi @physimo , i'm the author of #359 you mentionned, i'd like to work on that with you, if you ok to share.
@giladgd please provide a simple working example of shiftStrategy, with store full history somewhere (memory or file, or rag) then sumarize when context reach à percent of full contextwindow, and reinject summarization to allow continuing conversation

[giladgd]

I'll release a new version in the next week or so that'll include updated documentation with an example of implementing a custom context shift strategy.
That example won't include a summarization of the existing context, as that profoundly depends on your use case, but it'll give you an idea of how to implement it yourself.

[giladgd]

I've released a new version with an updated documentation that includes a guide on how to create your own custom chat context shift strategy.