-
Notifications
You must be signed in to change notification settings - Fork 25
Add learn-from-code-review skill #87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
44ec0ae
Add learn-from-code-review skill
openhands-agent d80dd04
Address review feedback on learn-from-code-review skill
openhands-agent 5a2b6bb
Remove code review references from example outputs
openhands-agent 6ff378f
Prefer skills over AGENTS.md updates in output guidance
openhands-agent 1352d11
Clarify that no output is acceptable when no patterns emerge
openhands-agent File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Learn from Code Review | ||
|
|
||
| Distill code review feedback from GitHub pull requests into reusable skills and repository guidelines. | ||
|
|
||
| ## What It Does | ||
|
|
||
| This skill analyzes PR review comments from a repository and extracts recurring patterns of feedback. These patterns are then transformed into: | ||
|
|
||
| - **Repository-specific skills** (`.openhands/skills/`) for domain-specific patterns | ||
| - **AGENTS.md updates** for general coding conventions and best practices | ||
|
|
||
| ## When to Use | ||
|
|
||
| - "Learn from our code reviews" | ||
| - "Distill PR feedback into guidelines" | ||
| - "What patterns do reviewers keep pointing out?" | ||
| - "Generate coding standards from review history" | ||
| - Running `/learn-from-reviews` | ||
|
|
||
| ## Requirements | ||
|
|
||
| - `GITHUB_TOKEN` environment variable | ||
| - GitHub CLI (`gh`) available | ||
|
|
||
| ## Example | ||
|
|
||
| ``` | ||
| User: Learn from our code reviews over the past month | ||
|
|
||
| Agent: I'll analyze recent PR review comments and distill them into actionable guidelines. | ||
|
|
||
| [Analyzes 25 merged PRs with 150 review comments] | ||
| [Filters out bot comments and low-signal responses] | ||
| [Identifies 4 recurring patterns] | ||
|
|
||
| Found the following patterns from code review feedback: | ||
|
|
||
| 1. Error Handling (12 comments) | ||
| - Always include context when logging errors | ||
| - Use structured error responses in APIs | ||
|
|
||
| 2. Testing (8 comments) | ||
| - Add edge case tests for validation logic | ||
| - Mock external services consistently | ||
|
|
||
| 3. Database Queries (6 comments) | ||
| - Use parameterized queries exclusively | ||
| - Add appropriate indexes for new queries | ||
|
|
||
| I'll create a draft PR with: | ||
| - New skill: `.openhands/skills/error-handling/SKILL.md` | ||
| - Updates to `AGENTS.md` with testing and database guidelines | ||
| ``` | ||
|
|
||
| ## Output | ||
|
|
||
| The skill generates a draft PR containing proposed changes based on the analysis. Human review is expected before merging. | ||
|
|
||
| ## Related Skills | ||
|
|
||
| - `github-pr-review` - Post structured code reviews | ||
| - `code-review` - Perform code reviews | ||
| - `skill-creator` - Create new skills manually | ||
| - `agent-memory` - Persist repository knowledge | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,186 @@ | ||
| --- | ||
| name: learn-from-code-review | ||
| description: Distill code review feedback from GitHub PRs into reusable skills and guidelines. This skill should be used when users ask to "learn from code reviews", "distill PR feedback", "improve coding standards", "extract learnings from reviews", or want to generate skills/guidelines from historical review comments. | ||
| triggers: | ||
| - /learn-from-reviews | ||
| - learn from code review | ||
| - distill reviews | ||
| --- | ||
|
|
||
| # Learn from Code Review | ||
|
|
||
| Analyze code review comments from GitHub pull requests and distill them into reusable skills or repository guidelines that improve future code quality. | ||
|
|
||
| ## Overview | ||
|
|
||
| Code review feedback contains valuable institutional knowledge that often gets buried across hundreds of PRs. This skill extracts meaningful patterns from review comments and transforms them into: | ||
|
|
||
neubig marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| 1. **Repository-specific skills** - Placed in `.openhands/skills/` for domain-specific patterns | ||
| 2. **AGENTS.md guidelines** - Overall repository conventions and best practices | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - `GITHUB_TOKEN` environment variable must be set | ||
| - GitHub CLI (`gh`) should be available | ||
|
|
||
| ## Workflow | ||
|
|
||
| ### Step 1: Identify Target Repository | ||
|
|
||
| Determine the repository to analyze: | ||
|
|
||
| ```bash | ||
| # Get current repo info | ||
| gh repo view --json nameWithOwner -q '.nameWithOwner' | ||
| ``` | ||
|
|
||
| If not in a repository, ask the user which repository to analyze. | ||
|
|
||
| ### Step 2: Fetch Review Comments | ||
|
|
||
| Retrieve PR review comments from the repository: | ||
|
|
||
| ```bash | ||
| # Fetch merged PRs from the last 30 days (adjustable) | ||
| gh pr list --repo {owner}/{repo} \ | ||
| --state merged \ | ||
| --limit 50 \ | ||
| --json number,title,mergedAt | ||
|
|
||
| # For each PR, fetch review comments | ||
| gh api repos/{owner}/{repo}/pulls/{pr_number}/comments \ | ||
| --jq '.[] | {body: .body, path: .path, user: .user.login, created_at: .created_at}' | ||
|
|
||
| # Also fetch review-level comments (not tied to specific lines) | ||
| gh api repos/{owner}/{repo}/pulls/{pr_number}/reviews \ | ||
| --jq '.[] | select(.body != "") | {body: .body, user: .user.login, state: .state}' | ||
enyst marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ``` | ||
|
|
||
| ### Step 3: Filter and Categorize Comments | ||
|
|
||
| Apply noise filtering to keep only meaningful feedback: | ||
|
|
||
| **Exclude:** | ||
| - Bot comments (dependabot, copilot, github-actions, etc.) | ||
neubig marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - Low-signal responses ("LGTM", "+1", "looks good", "thanks", "nice") | ||
| - Comments shorter than 30 characters | ||
| - Auto-generated comments (CI status, coverage reports) | ||
|
|
||
| **Categorize remaining comments by:** | ||
| - Security concerns | ||
| - Performance patterns | ||
| - Code style/conventions | ||
| - Architecture/design patterns | ||
| - Error handling | ||
| - Testing requirements | ||
| - Documentation standards | ||
|
|
||
| ### Step 4: Distill Patterns | ||
|
|
||
| For each category with sufficient examples (3+ similar comments), identify: | ||
|
|
||
neubig marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| 1. **The recurring issue** - What mistake or oversight keeps appearing | ||
| 2. **The desired pattern** - What reviewers consistently ask for | ||
| 3. **Example context** - Concrete before/after code snippets when available | ||
|
|
||
| ### Step 5: Generate Output | ||
|
|
||
| If clear, actionable patterns emerge, generate focused skill files. If no clear patterns emerge, report this to the user—it's fine to produce no output when the codebase already has strong conventions or when review comments don't cluster into recurring themes. | ||
|
|
||
| When creating skills, place them in `.openhands/skills/{domain-name}/SKILL.md`: | ||
|
|
||
| ```yaml | ||
| --- | ||
| name: database-queries | ||
| description: Database query patterns and best practices for this repository. | ||
| --- | ||
|
|
||
neubig marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # Database Query Guidelines | ||
|
|
||
| ### Always Use Parameterized Queries | ||
| [Pattern description with examples] | ||
|
|
||
| ### Connection Pool Management | ||
| [Pattern description with examples] | ||
| ``` | ||
|
|
||
| Prefer skills over AGENTS.md updates, since AGENTS.md typically already contains general coding guidelines. | ||
|
|
||
| ### Step 6: Create Draft PR (if applicable) | ||
|
|
||
| Use the `create_pr` tool to open a draft PR with the proposed changes. The PR description should include: | ||
| - Number of PRs analyzed | ||
| - Number of comments processed | ||
| - Categories of patterns found | ||
| - List of proposed changes (new skills and/or AGENTS.md updates) | ||
|
|
||
| ## Example Output | ||
|
|
||
| ### Sample Skill: API Error Handling | ||
|
|
||
| ```yaml | ||
| --- | ||
| name: api-error-handling | ||
| description: API error handling patterns for this repository. | ||
| --- | ||
|
|
||
| # API Error Handling | ||
|
|
||
| ## Always Return Structured Errors | ||
|
|
||
| ❌ Avoid: | ||
| ```python | ||
| return {"error": str(e)} | ||
| ``` | ||
|
|
||
| ✅ Prefer: | ||
| ```python | ||
| return { | ||
| "error": { | ||
| "code": "VALIDATION_ERROR", | ||
| "message": "Invalid input", | ||
| "details": {"field": "email", "reason": "Invalid format"} | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Log Before Returning Errors | ||
|
|
||
| ```python | ||
| logger.error(f"API error in {endpoint}: {e}", exc_info=True) | ||
| return error_response(e) | ||
| ``` | ||
| ``` | ||
|
|
||
| ## Defaults | ||
|
|
||
| This workflow analyzes PRs from the past 30 days by default. | ||
|
|
||
| ## Best Practices | ||
|
|
||
| 1. **Run periodically** - Schedule monthly or quarterly to capture evolving patterns | ||
| 2. **Review before merging** - Generated content is a draft; human review is essential | ||
| 3. **Iterate** - Refine patterns based on team feedback | ||
| 4. **Avoid duplication** - Check existing AGENTS.md and skills before adding | ||
| 5. **Cite sources** - Reference PR numbers when documenting patterns | ||
neubig marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## Error Handling | ||
|
|
||
| Handle these common edge cases gracefully: | ||
|
|
||
| - **Repository has few PRs**: If fewer than 10 merged PRs exist in the timeframe, inform the user that there may not be enough data to identify patterns. Proceed with analysis but note the limited sample size. | ||
| - **No patterns emerge**: When comments don't cluster into recurring themes (common for well-established codebases), report this to the user and suggest either expanding the time range or that the codebase may already have strong conventions. | ||
| - **Token lacks repository access**: If the GitHub API returns 403/404, explain that the token may not have access to the repository and suggest checking token permissions. | ||
| - **`gh` CLI unavailable**: Fall back to direct GitHub API calls using `curl` with `$GITHUB_TOKEN`, or inform the user that `gh` needs to be installed. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - Only analyzes accessible repositories (requires appropriate permissions) | ||
| - Cannot capture verbal feedback from pair programming or meetings | ||
| - Patterns may reflect individual reviewer preferences vs. team consensus | ||
| - Historical comments may reference outdated code patterns | ||
|
|
||
| ## Additional Resources | ||
|
|
||
| For posting structured code reviews, see the `github-pr-review` skill. | ||
| For creating new skills, see the `skill-creator` skill. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.