-
Notifications
You must be signed in to change notification settings - Fork 2.6k
fix: enhance null edit detection for Gemini models #7362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
- Add detailed error messages when search and replace content are identical - Provide clear guidance on common causes (model hallucination, copy/paste errors) - Add comprehensive tests for null edit detection - Help AI models (especially Gemini) avoid producing phantom edits Fixes #7360
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewing my own code is like debugging in a mirror - everything looks backwards but the bugs are still mine.
| success: false, | ||
| error: | ||
| `Search and replace content are identical - no changes would be made\n\n` + | ||
| `NULL EDIT DETECTED: Search and replace content are identical - no changes would be made\n\n` + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider extracting this error message to a shared constant since it's duplicated in multi-file-search-replace.ts (lines 518-528). This would ensure consistency and make future updates easier.
Also, while the error message is comprehensive, we might want to add a test case for when search and replace blocks differ only in whitespace type (tabs vs spaces) to ensure we're not triggering false positives.
| } | ||
| }) | ||
|
|
||
| it("should not trigger null edit detection for legitimate empty replacements (deletions)", async () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great test coverage! Consider adding a comment here explaining why legitimate deletions (empty replace blocks) should not trigger null edit detection. This distinction is important for future maintainers to understand the design decision.
| } | ||
| }) | ||
|
|
||
| it("should not trigger for actual changes even if similar", async () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we add an edge case test for when search and replace blocks have different whitespace but are otherwise identical? This would help document the expected behavior when whitespace types differ (tabs vs spaces).
|
Just adding another data point - I'm seeing this on self-hosted llama-4-maverick too, so likely not model or provider specific |
Summary
This PR addresses Issue #7360 where Gemini models (2.5 Pro, 2.5 Flash, 2.5 Flash:Thinking) produce "phantom" edits that appear to make changes but actually result in zero character modifications.
Problem
The user reported that Gemini models would get into non-productive loops where they would claim to be making edits but the search and replace blocks would contain identical content, resulting in no actual changes to the files.
Solution
Enhanced the diff validation in both diff strategies to:
Changes
multi-search-replace.tsandmulti-file-search-replace.tsnull-edit-detection.spec.tsTesting
Impact
This change will help AI models, especially Gemini, avoid producing phantom edits and provide clear guidance when null edits are detected, improving the overall user experience.
Fixes #7360
Important
Enhances null edit detection for Gemini models by improving error messages and adding comprehensive test coverage.
multi-search-replace.tsandmulti-file-search-replace.tsto identify identical search and replace content.null-edit-detection.spec.tsto test null edit detection, ensuring legitimate deletions are not flagged.This description was created by
for 4a114a2. You can customize this summary. It will automatically update as commits are pushed.