Skip to content

PR review agent: avoid approving eval-risk behavior changes #2246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
enyst merged 2 commits into from
Feb 28, 2026
Merged

PR review agent: avoid approving eval-risk behavior changes #2246

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2da95a9
chore(pr-review): avoid approving eval-risk changes
openhands-agent Feb 28, 2026
a1a0683
Update prompt.py
enyst Feb 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions examples/03_github_workflows/02_pr_review/prompt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,21 @@
PROMPT = """{skill_trigger}
/github-pr-review

When posting a review, keep the review body brief unless your active review instructions
require a longer structured format.
When posting a review, keep the review body brief unless your active review
instructions require a longer structured format.

## Review decision policy (eval / benchmark risk)

You MAY approve clearly low-risk changes (docs, typo fixes, formatting, or
pure refactors with no behavior changes).

Do NOT submit an APPROVE review when the PR changes agent behavior or anything
that affects benchmark/evaluation performance.
Examples include: prompt templates, tool calling/execution, planning/loop logic,
memory/condenser behavior, terminal/stdin/stdout handling, or evaluation harness code.

If a PR is in this category (or you are uncertain), leave a COMMENTED review and
explicitly flag it for a human maintainer to decide after running lightweight evals.
Copy link
Copy Markdown
Collaborator

@xingyaoww xingyaoww Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@enyst I think we should not put the prompt in this file, since it will affect all the repos using the codereview action, right? i think we should put it into a repository specific .agents/code-review.md

Copy link
Copy Markdown
Collaborator Author

@enyst enyst Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OpenHands Let's fix this

Copy link
Copy Markdown

@openhands-ai openhands-ai bot Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on it! enyst can track my progress at all-hands.dev

Copy link
Copy Markdown
Collaborator Author

@enyst enyst Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course, oops

Copy link
Copy Markdown

@openhands-ai openhands-ai bot Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of work completed

  • Identified that the “eval / benchmark risk” review decision policy was added to examples/03_github_workflows/02_pr_review/prompt.py, which is used by the PR-review GitHub Action and therefore would affect all downstream repos that use the action.
  • Implemented the requested fix by moving that policy out of the shared example prompt and into this repository’s repo-specific review guidance at .agents/skills/custom-codereview-guide.md.

Changes made (concise)

  • examples/03_github_workflows/02_pr_review/prompt.py
    • Removed the eval/benchmark-risk “do not APPROVE…” section so the action’s base prompt remains generic for all repos.
  • .agents/skills/custom-codereview-guide.md
    • Added a new “Review decision policy (eval / benchmark risk)” section that instructs reviewers to COMMENT instead of APPROVE when changes could plausibly affect benchmark/eval performance.

Verification / compliance checklist

  • Addressed the reviewer concern about global impact by relocating policy to repo-specific rules.
  • Kept changes minimal (only 2 files, no unrelated edits).
  • Ran uv run pre-commit run --files ... on the modified files (passed).
  • Commit message included Co-authored-by: openhands <openhands@all-hands.dev>.

Delivery status


Review the PR changes below and identify issues that need to be addressed.

Expand Down
Loading