Interplay with VectorCode?

[Davidyz]

I'm loving the idea of being able to save the chat history, and it feels like it's just one step away from implementing a long term memory for codecompanion. I haven't got any concrete plans yet, but I've got some initial ideas that we can try:

1. treat dir_to_save as an ordinary directory and vectorise the full chat files;
2. reuse some of the code for title generation to generate a summary for each chat history (it'll be more convenient if they're stored in a separate file), and vectorise the summary instead;
3. similar to 2, but generate a list of keywords instead of a summary in natural language.

Regardless of which method we choose, we should probably make sure that the working directory is in the metadata too, because it'll be related to which project the chat is about.

I'm personally leaning toward 2, but I'm also open to suggestions and ideas!

[ravitemer]

Thanks @Davidyz. Love this idea.

reuse some of the code for title generation to generate a summary for each chat history (it'll be more convenient if they're stored in a separate file), and vectorise the summary instead;

Yes I too prefer this option.

Regardless of which method we choose, we should probably make sure that the working directory is in the metadata too, because it'll be related to which project the chat is about.

We can do that.

How do you think we can use this long term memory, from a user's perspective? Do you need something from my side to make this work?

[ravitemer]

I wouldn't want it to limit search results purely based on where the current root is.

I feel the same.

I haven't started working on this yet, so I'm not quite sure what I'd need, but the working directory would be useful if the user needs the LLM to keep using the VectorCode tool.

@Davidyz you mentioned that vectorcode tool needs the pwd. Is it possible for the LLM to pass the pwd while calling the tool based on the chat history (falling back to vim.fn.getpwd())?

[Davidyz]

@Davidyz you mentioned that vectorcode tool needs the pwd.

On a second thought, I can just put the project root in the system prompt of the Vectorcode tool (which is already implemented). In this case pwd in the metadata won't be necessary.

[Davidyz]

In fact, when the summary feature is done, could you implement a picker/vim.ui.select interface for them (something like codecompanion /file or /buffer)? This should be easier to implement than the VectorCode counterpart (I want to make it automatic so that users won't need to manually index the memories, but that's gonna take longer). We can use this to investigate how well the LLM understand the summaries, and maybe fine-tune the instructions for summary generation. And if this works well, we can confidently move forward with this using vector search and other fancy stuff.

[ravitemer]

Sure! Will implement a proof of concept summary implementation and a picker for it. On first thought, I want to make this optional so that user decides at which point of the conversation he wants to generate a summary? We can update the title of the chat to show whether it has summary associated with it (and last generated time). And with some keymap like ghs or something we open the picker. This should give us as you said a base to tinker the prompt or maybe take a totally different approach like 1. treat dir_to_save as an ordinary directory and vectorise the full chat files;

Ideally, we should generate the summaries automatic in the future maybe after getting a response from the LLM. However, generating the summaries comes with it's own disadvantages mainly relating to token consumption once in production. Once we test and see how the summary generation works, we can think of another approach. I will open a PR for this soon.

[Davidyz]

make this optional so that user decides at which point of the conversation he wants to generate a summary?

I think this is sufficient for a proof-of-concept feature.

However, generating the summaries comes with it's own disadvantages mainly relating to token consumption once in production.

But we might be able to dynamically swap old messages in a long conversation with their summary 👀 Could be very helpful when the conversation gets longer.