Skip to content

Conversation

@roomote
Copy link
Contributor

@roomote roomote bot commented Aug 23, 2025

Summary

This PR addresses Issue #7360 where Gemini models (2.5 Pro, 2.5 Flash, 2.5 Flash:Thinking) produce "phantom" edits that appear to make changes but actually result in zero character modifications.

Problem

The user reported that Gemini models would get into non-productive loops where they would claim to be making edits but the search and replace blocks would contain identical content, resulting in no actual changes to the files.

Solution

Enhanced the diff validation in both diff strategies to:

  1. Detect when search and replace content are identical (null edits)
  2. Provide clear, detailed error messages specifically mentioning this is a common Gemini issue
  3. Guide users with actionable steps to fix the problem
  4. List common causes (model hallucination, copy/paste errors, missing modifications)

Changes

  • Enhanced error messages in multi-search-replace.ts and multi-file-search-replace.ts
  • Added comprehensive test coverage in null-edit-detection.spec.ts
  • Tests verify null edit detection while ensuring legitimate deletions still work

Testing

  • ✅ All new tests passing (5/5)
  • ✅ No regression in existing tests (60/60 passing)
  • ✅ Linting and type checking pass

Impact

This change will help AI models, especially Gemini, avoid producing phantom edits and provide clear guidance when null edits are detected, improving the overall user experience.

Fixes #7360


Important

Enhances null edit detection for Gemini models by improving error messages and adding comprehensive test coverage.

  • Behavior:
    • Enhanced null edit detection in multi-search-replace.ts and multi-file-search-replace.ts to identify identical search and replace content.
    • Provides detailed error messages for null edits, highlighting common causes and solutions.
  • Testing:
    • Added null-edit-detection.spec.ts to test null edit detection, ensuring legitimate deletions are not flagged.
    • Tests cover scenarios like identical search/replace, multi-block diffs, and line number handling.
  • Impact:
    • Improves user experience by preventing non-productive loops in Gemini models and guiding users to resolve null edits.

This description was created by Ellipsis for 4a114a2. You can customize this summary. It will automatically update as commits are pushed.

- Add detailed error messages when search and replace content are identical
- Provide clear guidance on common causes (model hallucination, copy/paste errors)
- Add comprehensive tests for null edit detection
- Help AI models (especially Gemini) avoid producing phantom edits

Fixes #7360
@roomote roomote bot requested review from cte, jr and mrubens as code owners August 23, 2025 21:44
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Aug 23, 2025
@hannesrudolph hannesrudolph added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Aug 23, 2025
Copy link
Contributor Author

@roomote roomote bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing my own code is like debugging in a mirror - everything looks backwards but the bugs are still mine.

success: false,
error:
`Search and replace content are identical - no changes would be made\n\n` +
`NULL EDIT DETECTED: Search and replace content are identical - no changes would be made\n\n` +
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider extracting this error message to a shared constant since it's duplicated in multi-file-search-replace.ts (lines 518-528). This would ensure consistency and make future updates easier.

Also, while the error message is comprehensive, we might want to add a test case for when search and replace blocks differ only in whitespace type (tabs vs spaces) to ensure we're not triggering false positives.

}
})

it("should not trigger null edit detection for legitimate empty replacements (deletions)", async () => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great test coverage! Consider adding a comment here explaining why legitimate deletions (empty replace blocks) should not trigger null edit detection. This distinction is important for future maintainers to understand the design decision.

}
})

it("should not trigger for actual changes even if similar", async () => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add an edge case test for when search and replace blocks have different whitespace but are otherwise identical? This would help document the expected behavior when whitespace types differ (tabs vs spaces).

@daniel-lxs daniel-lxs moved this from Triage to Issue [In Progress] in Roo Code Roadmap Aug 26, 2025
@daniel-lxs daniel-lxs moved this from Issue [In Progress] to PR [Needs Prelim Review] in Roo Code Roadmap Aug 26, 2025
@hannesrudolph hannesrudolph added PR - Needs Preliminary Review and removed Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. labels Aug 26, 2025
@bstrdsmkr
Copy link

Just adding another data point - I'm seeing this on self-hosted llama-4-maverick too, so likely not model or provider specific

@daniel-lxs
Copy link
Member

#7360 (comment)

@daniel-lxs daniel-lxs closed this Aug 27, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Aug 27, 2025
@github-project-automation github-project-automation bot moved this from PR [Needs Prelim Review] to Done in Roo Code Roadmap Aug 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working PR - Needs Preliminary Review size:L This PR changes 100-499 lines, ignoring generated files.

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Roo-Code using Gemini multiple null function edits

5 participants