feat: add re-read after edit experiment for automatic review suggestions #8928
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements issue #8927 - Add option to suggest reviewing changes after file edits. - Added RE_READ_AFTER_EDIT experiment configuration - Updated all file editing tools to suggest review when experiment is enabled - Added UI setting with English translation - Added tests for the new experiment When enabled, the AI will suggest using the read_file tool to review changes after editing files, helping catch errors that might be introduced during editing.
dosubot
bot
added
size:M
Review CompleteI've reviewed the changes in this PR and found 1 issue that should be addressed: Issues Found
|
2 tasks
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
Comment on lines
+687
to
+692
| // Count how many files were successfully edited | ||
| const editedFiles = operationResults.filter((op) => op.status === "approved").length | ||
| const reReadSuggestion = | ||
| isReReadAfterEditEnabled && editedFiles > 0 | ||
| ? `\n\n<review_suggestion>${editedFiles === 1 ? "The file has" : `${editedFiles} files have`} been edited. Consider using the read_file tool to review the changes and ensure they are correct and complete.</review_suggestion>` | ||
| : "" |
There was a problem hiding this comment.
The review suggestion count may be inaccurate. When a file is approved but all its diffs fail to apply (successCount === 0 at line 494), the code continues without saving changes (line 514), but the file remains with status "approved". Later at line 688, the code counts all "approved" files for the review suggestion, incorrectly including files that were approved but never actually edited. This could result in suggestions like "2 files have been edited" when only 1 file was successfully modified.
hannesrudolph
added
the
Issue/PR - Triage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements #8927 - Add option to suggest reviewing changes after file edits.
Summary
Adds an experimental feature that prompts the AI to review changes after editing files, helping catch errors that might be introduced during editing.
Changes
RE_READ_AFTER_EDITexperiment configuration to types and experiments systemapplyDiffTool.ts- Single file diff applicationsmultiApplyDiffTool.ts- Multi-file diff applicationswriteToFileTool.ts- File writesinsertContentTool.ts- Content insertionsearchAndReplaceTool.ts- Search and replace operationsHow it works
When the experiment is enabled, after any file editing operation, the AI will receive a
<review_suggestion>message suggesting to use theread_filetool to review the changes. The suggestions are contextual (e.g., "The file has been edited" vs "2 files have been edited").Trade-offs
As mentioned in the issue, this feature will:
This is why it's implemented as an opt-in experiment, allowing users to decide based on their needs.
Testing
Closes #8927
Important
Introduces
RE_READ_AFTER_EDITexperiment to suggest reviewing changes after file edits, with updates to tools, configuration, and tests.RE_READ_AFTER_EDITexperiment to suggest reviewing changes after file edits.applyDiffTool.ts,multiApplyDiffTool.ts,writeToFileTool.ts,insertContentTool.ts, andsearchAndReplaceTool.tsto include review suggestions when the experiment is enabled.reReadAfterEdittoexperiment.tsandexperiments.ts.RE_READ_AFTER_EDITinsettings.json.experiments-reReadAfterEdit.spec.tsfor testing the new experiment.experiments.spec.tsandExtensionStateContext.spec.tsxto include the new experiment.This description was created by
for f86af76. You can customize this summary. It will automatically update as commits are pushed.