[Enhancement]: Automatic context compaction

[nikita-vanyasin]

What features would you like to see added?

When large chunks of text are returned from the MCP tool, it is very easy to consume whole context of the model. It will just stop generating next tokens.

One of the approach to tackle that is to automatically compact the context. This is what Claude Code does when you are approaching the limits.

More details

Prompt for compacting the conversation: https://www.reddit.com/r/ClaudeAI/comments/1jr52qj/here_is_claude_codes_compact_prompt/

A bit of theory and implementation example in Forge

Which components are impacted by your request?

General

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Related

#1179

[danny-avila]

Thank you for your feature request about context compaction. I want to clarify that we already have a similar feature in place, but there's definitely room for improvement based on the Forge implementation you shared.

Our system already intelligently manages conversations when they get too long by selectively removing older messages while preserving critical information. This helps prevent hitting token limits while maintaining conversation quality.

The key difference between our current approach and what Forge implements is:

• Our current system removes or "prunes" older messages when conversations get too long
• Forge's approach summarizes parts of the conversation instead of removing them

Potential Improvements We Could Make

Based on the article and code you shared, we could enhance our system to:

1. Create summaries instead of removing content: When tool outputs return large chunks of text, we could generate concise summaries that preserve the key information

2. Add more intelligent selection: Instead of just pruning older messages, we could identify specific patterns (like sequences of tool calls and their results) that are good candidates for summarization

3. Provide more control options: We could add settings to control:

   • When summarization happens (based on token count, number of messages, etc.)
   • Which messages to preserve completely
   • How detailed the summaries should be

4. Use dedicated models for summarization: Like Forge does, we could use smaller, faster models specifically for creating summaries

It will just stop generating next tokens.

If you're experiencing this, you're reaching the model's upper limit beyond the 90% allotted by the system (which is not perfectly accurate but "good enough").

I'm not sure how critical this might be to implement because on the next turn the model should then get back at least 10% of its limit. This would still be worthwhile exploring so I thought it'd be good for me to outline.

As mentioned before, control options could help net improvements without the complexity of adding summaries into the mix, because we could control the context window more, especially once a certain threshold is met, etc.

[nikita-vanyasin]

Thank you for detailed answer @danny-avila !

by selectively removing older messages while preserving critical information.

Do you mind sharing more details how it is implemented / link to source code?

Is it something I can configure?

I've found "Max Context Tokens" field in model parameters. Is it related? When I try to set it for Bedrock + Anthropic Claude combination, it fails during the first chat message:

Something went wrong. Here's the specific error message we encountered: An error occurred while processing the request: The model returned the following errors: maxContextTokens: Extra inputs are not permitted

Thank you!

[avx-rodmans]

Plus one on this