feat: implement read_file history deduplication

[roomote]

Related GitHub Issue

Closes: #6279

Roo Code Task Context (Optional)

Description

This PR implements a read_file history deduplication feature to prevent duplicate file reads from accumulating in the conversation history. The implementation:

• Adds a new experimental feature flag READ_FILE_DEDUPLICATION that defaults to disabled
• Implements a deduplicateReadFileHistory method in the Task class that removes older duplicate read_file results while preserving the most recent ones
• Preserves a 5-minute cache window to avoid modifying recent messages
• Supports both single-file and multi-file read operations in the new XML format
• Maintains backward compatibility with legacy read_file result formats
• Only removes duplicate read_file content blocks while preserving all other message content

Test Procedure

1. Enable the readFileDeduplication experimental feature in settings
2. Use Roo Code to read the same file multiple times during a conversation
3. Verify that older duplicate read_file results are removed from the conversation history
4. Verify that recent reads (within 5 minutes) are preserved
5. Run the unit tests: cd src && npx vitest run core/task/__tests__/Task.spec.ts

All unit tests have been added to cover:

• Feature toggle behavior
• Basic deduplication functionality
• Cache window preservation
• Multi-file support
• Content preservation for non-read_file blocks
• Legacy format compatibility

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Testing: New and/or updated tests have been added to cover my changes (if applicable).
• Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Screenshots / Videos

Not applicable - this is a backend feature with no UI changes.

Documentation Updates

• No documentation updates are required.

The feature is experimental and disabled by default. When it's ready for general use, documentation should be updated to explain the feature.

Additional Notes

This implementation carefully preserves message structure and only removes duplicate read_file content blocks. It includes comprehensive error handling and maintains backward compatibility with older message formats.

Get in Touch

---

Important

Introduces read_file history deduplication feature with a new feature flag, implementation in Task.ts, and comprehensive test coverage.

• Feature:
  • Adds READ_FILE_DEDUPLICATION feature flag in experiment.ts.
  • Implements deduplicateReadFileHistory() in Task.ts to remove older duplicate read_file results, preserving recent ones.
  • Supports both XML and legacy formats, handles single and multi-file reads.
• Tests:
  • Adds tests for deduplication logic in Task.spec.ts.
  • Mocks deduplicateReadFileHistory in readFileTool.spec.ts.
  • Updates experiments.spec.ts to include READ_FILE_DEDUPLICATION.
• Misc:
  • Calls deduplicateReadFileHistory() in readFileTool.ts after successful reads.

^{This description was created by for 0fe3571. You can customize this summary. It will automatically update as commits are pushed.}

[hannesrudolph]

@roomote-agent does this only trigger the deduplication on reads
Or in every message? It should only be on reads.

[roomote]

Hi @hannesrudolph, I see your question about the deduplication trigger. Let me investigate the implementation to confirm when exactly the deduplication is triggered and get back to you with a detailed answer.

[roomote]

Hi @hannesrudolph, Ive investigated and fixed the issue.

You were correct - the deduplication was being called in the error handler instead of only on successful reads. Ive moved the deduplicateReadFileHistory() call from the catch block to the success path (right after pushing the file results).

The deduplication now only triggers after successful read_file operations, not on every message. Ive also updated the test mocks to include the deduplicateReadFileHistory method and all tests are passing.

The fix has been pushed to the PR branch and CI checks are currently running.

[ellipsis-dev]

Good: The new deduplicateReadFileHistory mock is added to cline. Consider adding an assertion in one of the tests to ensure this function is called exactly once during a successful read_file operation, to cover that behavior explicitly.

[hannesrudolph]

@roomote-agent Run pr-reviewer on this

[roomote]

Hi @hannesrudolph, I see your request to run pr-reviewer on this PR. Let me switch to PR Reviewer mode and perform a comprehensive review of the read_file history deduplication implementation.

[roomote]

PR Review: feat: implement read_file history deduplication (#6280)

Executive Summary

This PR implements a read_file history deduplication feature that removes older duplicate file reads from the conversation history while preserving recent ones. The implementation is well-structured with comprehensive test coverage and follows established patterns in the codebase.

✅ Strengths

1. Feature Flag Implementation: Properly implemented as an experimental feature that defaults to disabled, following the established pattern used by other experiments like MULTI_FILE_APPLY_DIFF and POWER_STEERING.

2. Comprehensive Test Coverage: Excellent test coverage including:

   • Feature toggle behavior
   • Basic deduplication functionality
   • Cache window preservation (5-minute window)
   • Multi-file support
   • Content preservation for non-read_file blocks
   • Legacy format compatibility

3. Backward Compatibility: Maintains support for both the new XML format and legacy read_file result formats.

4. Safe Implementation:

   • Only processes user messages (skips assistant messages)
   • Preserves messages within a 5-minute cache window
   • Only removes duplicate read_file content blocks while preserving other content
   • Proper error handling with try-catch blocks

5. Correct Integration Point: The deduplication is called after successful file reads in readFileTool.ts (line 594), which addresses the concern raised in the PR comments about only triggering on reads.

🔍 Code Quality Observations

Pattern Consistency

The implementation follows established patterns:

• Uses experiments.isEnabled() pattern consistent with other experimental features
• Properly integrated into the experiment system with entries in experiment.ts, experiments.ts, and test files
• Follows the same structure as other Task methods

Implementation Details

The deduplicateReadFileHistory method in Task.ts:

• Efficiently tracks seen files using a Set
• Iterates through history in reverse order (newest to oldest)
• Uses regex matching to extract file paths from both XML and legacy formats
• Properly updates the conversation history and saves it

📝 Minor Suggestions

1. Consider Adding Metrics: It might be useful to track how often deduplication occurs and how many messages are removed for monitoring the feature's effectiveness.

2. Documentation: While the code is well-commented, consider adding a brief explanation in the PR description about why the 5-minute cache window was chosen.

3. Performance Consideration: For very large conversation histories, the reverse iteration and regex matching could be optimized, but this is likely not a concern for typical usage.

✅ Verification

• The implementation correctly addresses the issue raised in PR comments about only triggering on successful reads
• The feature is properly gated behind an experimental flag
• All tests are comprehensive and cover edge cases
• The code follows TypeScript best practices and maintains type safety

Conclusion

This is a well-implemented feature that addresses a real need for preventing duplicate file reads in conversation history. The code is clean, well-tested, and follows established patterns in the codebase. The experimental flag approach allows for safe rollout and testing.

Recommendation: ✅ Approve - This PR is ready for merge.

[hannesrudolph]

Update: Implemented Proactive File Caching

After further testing, I discovered that the initial reactive approach wasn't working as expected. The deduplication was happening after files were already read and added to the context, which didn't reduce the context size.

Changes in this update:

1. Switched to proactive caching: Instead of removing duplicates after reading, we now check for cached content before reading files
2. Added method: This method checks if a file was recently read (within the cache window) and returns the cached content
3. **Modified **: Now uses to check for cached content before reading from disk
4. Added notice when using cache: When cached content is used, a notice is added to inform that the content is from cache

Benefits:

• Actually prevents redundant file reads from increasing context size
• More efficient as it avoids disk I/O when possible
• Provides transparency when cached content is used

All tests are passing and the feature now works as intended. The cache time window remains configurable through the settings UI.