Compacting selected blocks in history #214
Description
As we are going to support "user prompts menus" I got an idea inspired by Droid.
Droid automatically compacts parts in chat history, which helps maintain very long sessions.
I would take a manual approach where you can tag two prompts (by clicking on a new "Anchor compact" button), and then a new "Compact" button would appear under both prompts. When you click on it, the LLM will compact the anchored block and use it in its history.
What the user sees in the chat remains unchanged. This is very powerful way to maintain long sessions in my opinion.
NOTE: There is still something I need to figure out, and that is how to deal with this compact related to caching. So consider this ticket to be WIP.
To upvote this issue, give it a thumbs up. See this list for the most upvoted issues.