Make Compact safer

[zikajk]

Describe the bug
I called compact, Gemini decided to use clojure_scratchpad instead, so i canceled it and ECA deleted my full conversation.
We should have some check that will make sure LLM really called the tool as the last step before we delete the conversation

Expected behavior
Do not clear conversation until we are 100% confident that compact was successful.

Maybe it would be better to prompt LLM (instead of calling the tool) on background and return only the status / compacted block of text to the chat window so there is no risk that model will call the wrong tool etc.

To upvote this issue, give it a thumbs up. See this list for the most upvoted issues.

[ericdallo]

Hum, that's bad, but yes, I believe only cleaning the conversation after summary worked would be a nice thing

[zikajk]

What do you think about ditching the tool approach? I see some risks with current approach:
a) LLM can call it based on your prompt where you don't want it to.
b) LLM might not use it when it should (as experienced with Google Gemini 3 Flash)
c) User can disable it by a mistake (not yet, but with enabledTools it can happen)

Is it worth the work? If not, then I think we should make sure the reported problem can't happen and we also make a and c case not possible.

[ericdallo]

What's the other approach if not using a tool call?
I feel like the tool solves some problems like give the power to LLM call it passing the specified args, following the tool description and so.

[zikajk]

OpenAI has an endpoint. Anthropic doesn't use Tool and it is part of SDK and so on.
This is a nice article, not fully relevant to the problem here though.
https://factory.ai/news/evaluating-compression

Tbh. I don't see an advantage in tool having a description or specified args as the compact is already some predefined prompt + optional user details (but it it is a questionable how they are really useful).

I see two reasons why tool can be bad:
a) LLM will run it without user asking for it (I've never seen it, so far so good).
b) LLM doesn't call it on compact (reproduced).

But in this ticket we should probably just never remove the chat if the existing tool doesn't succeed as it seems like a simpler fix.

[ericdallo]

yeah, I think we can experiment removing the tool leaving only the prompt if that proves to work with more LLMs and keep working with existing, I'm ok with that