Skip to content

Commit 406f82a

Browse files
neubigopenhands-agent
neubig
and
openhands-agent
authored
Add learn-from-code-review skill (#87)
* Add learn-from-code-review skill This skill distills code review feedback from GitHub PRs into reusable skills and repository guidelines. It analyzes review comments, filters out noise, identifies recurring patterns, and generates: - Repository-specific skills for domain patterns - AGENTS.md updates for general conventions Fixes #86 Co-authored-by: openhands <openhands@all-hands.dev> * Address review feedback on learn-from-code-review skill - Add Error Handling section documenting edge cases (few PRs, no patterns, token access issues, gh CLI unavailable) - Remove manual git commands, use create_pr tool instead - Replace Configuration Options table with simple defaults statement (30 days lookback) Co-authored-by: openhands <openhands@all-hands.dev> * Remove code review references from example outputs Focus generated skills on actionable guidelines rather than implementation details of how they were created. Co-authored-by: openhands <openhands@all-hands.dev> * Prefer skills over AGENTS.md updates in output guidance AGENTS.md typically already contains general coding guidelines, so generated output should focus on creating focused skill files. Co-authored-by: openhands <openhands@all-hands.dev> * Clarify that no output is acceptable when no patterns emerge Make it explicit that producing no skills is fine when the codebase already has strong conventions or comments don't cluster into recurring themes. Co-authored-by: openhands <openhands@all-hands.dev> --------- Co-authored-by: openhands <openhands@all-hands.dev>
1 parent 444a4c3 commit 406f82a

File tree

3 files changed

+263
-0
lines changed

3 files changed

+263
-0
lines changed

‎marketplaces/default.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,19 @@
238238
"cloud-native"
239239
]
240240
},
241+
{
242+
"name": "learn-from-code-review",
243+
"source": "./learn-from-code-review",
244+
"description": "Distill code review feedback from GitHub PRs into reusable skills and guidelines. Use when users ask to learn from code reviews, extract review patterns, or generate coding standards from historical PR feedback.",
245+
"category": "code-quality",
246+
"keywords": [
247+
"code-review",
248+
"learning",
249+
"skills",
250+
"guidelines",
251+
"pr-feedback"
252+
]
253+
},
241254
{
242255
"name": "notion",
243256
"source": "./notion",

‎skills/learn-from-code-review/README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Learn from Code Review
2+
3+
Distill code review feedback from GitHub pull requests into reusable skills and repository guidelines.
4+
5+
## What It Does
6+
7+
This skill analyzes PR review comments from a repository and extracts recurring patterns of feedback. These patterns are then transformed into:
8+
9+
- **Repository-specific skills** (`.openhands/skills/`) for domain-specific patterns
10+
- **AGENTS.md updates** for general coding conventions and best practices
11+
12+
## When to Use
13+
14+
- "Learn from our code reviews"
15+
- "Distill PR feedback into guidelines"
16+
- "What patterns do reviewers keep pointing out?"
17+
- "Generate coding standards from review history"
18+
- Running `/learn-from-reviews`
19+
20+
## Requirements
21+
22+
- `GITHUB_TOKEN` environment variable
23+
- GitHub CLI (`gh`) available
24+
25+
## Example
26+
27+
```
28+
User: Learn from our code reviews over the past month
29+
30+
Agent: I'll analyze recent PR review comments and distill them into actionable guidelines.
31+
32+
[Analyzes 25 merged PRs with 150 review comments]
33+
[Filters out bot comments and low-signal responses]
34+
[Identifies 4 recurring patterns]
35+
36+
Found the following patterns from code review feedback:
37+
38+
1. Error Handling (12 comments)
39+
- Always include context when logging errors
40+
- Use structured error responses in APIs
41+
42+
2. Testing (8 comments)
43+
- Add edge case tests for validation logic
44+
- Mock external services consistently
45+
46+
3. Database Queries (6 comments)
47+
- Use parameterized queries exclusively
48+
- Add appropriate indexes for new queries
49+
50+
I'll create a draft PR with:
51+
- New skill: `.openhands/skills/error-handling/SKILL.md`
52+
- Updates to `AGENTS.md` with testing and database guidelines
53+
```
54+
55+
## Output
56+
57+
The skill generates a draft PR containing proposed changes based on the analysis. Human review is expected before merging.
58+
59+
## Related Skills
60+
61+
- `github-pr-review` - Post structured code reviews
62+
- `code-review` - Perform code reviews
63+
- `skill-creator` - Create new skills manually
64+
- `agent-memory` - Persist repository knowledge

‎skills/learn-from-code-review/SKILL.md‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
---
2+
name: learn-from-code-review
3+
description: Distill code review feedback from GitHub PRs into reusable skills and guidelines. This skill should be used when users ask to "learn from code reviews", "distill PR feedback", "improve coding standards", "extract learnings from reviews", or want to generate skills/guidelines from historical review comments.
4+
triggers:
5+
- /learn-from-reviews
6+
- learn from code review
7+
- distill reviews
8+
---
9+
10+
# Learn from Code Review
11+
12+
Analyze code review comments from GitHub pull requests and distill them into reusable skills or repository guidelines that improve future code quality.
13+
14+
## Overview
15+
16+
Code review feedback contains valuable institutional knowledge that often gets buried across hundreds of PRs. This skill extracts meaningful patterns from review comments and transforms them into:
17+
18+
1. **Repository-specific skills** - Placed in `.openhands/skills/` for domain-specific patterns
19+
2. **AGENTS.md guidelines** - Overall repository conventions and best practices
20+
21+
## Prerequisites
22+
23+
- `GITHUB_TOKEN` environment variable must be set
24+
- GitHub CLI (`gh`) should be available
25+
26+
## Workflow
27+
28+
### Step 1: Identify Target Repository
29+
30+
Determine the repository to analyze:
31+
32+
```bash
33+
# Get current repo info
34+
gh repo view --json nameWithOwner -q '.nameWithOwner'
35+
```
36+
37+
If not in a repository, ask the user which repository to analyze.
38+
39+
### Step 2: Fetch Review Comments
40+
41+
Retrieve PR review comments from the repository:
42+
43+
```bash
44+
# Fetch merged PRs from the last 30 days (adjustable)
45+
gh pr list --repo {owner}/{repo} \
46+
--state merged \
47+
--limit 50 \
48+
--json number,title,mergedAt
49+
50+
# For each PR, fetch review comments
51+
gh api repos/{owner}/{repo}/pulls/{pr_number}/comments \
52+
--jq '.[] | {body: .body, path: .path, user: .user.login, created_at: .created_at}'
53+
54+
# Also fetch review-level comments (not tied to specific lines)
55+
gh api repos/{owner}/{repo}/pulls/{pr_number}/reviews \
56+
--jq '.[] | select(.body != "") | {body: .body, user: .user.login, state: .state}'
57+
```
58+
59+
### Step 3: Filter and Categorize Comments
60+
61+
Apply noise filtering to keep only meaningful feedback:
62+
63+
**Exclude:**
64+
- Bot comments (dependabot, copilot, github-actions, etc.)
65+
- Low-signal responses ("LGTM", "+1", "looks good", "thanks", "nice")
66+
- Comments shorter than 30 characters
67+
- Auto-generated comments (CI status, coverage reports)
68+
69+
**Categorize remaining comments by:**
70+
- Security concerns
71+
- Performance patterns
72+
- Code style/conventions
73+
- Architecture/design patterns
74+
- Error handling
75+
- Testing requirements
76+
- Documentation standards
77+
78+
### Step 4: Distill Patterns
79+
80+
For each category with sufficient examples (3+ similar comments), identify:
81+
82+
1. **The recurring issue** - What mistake or oversight keeps appearing
83+
2. **The desired pattern** - What reviewers consistently ask for
84+
3. **Example context** - Concrete before/after code snippets when available
85+
86+
### Step 5: Generate Output
87+
88+
If clear, actionable patterns emerge, generate focused skill files. If no clear patterns emerge, report this to the user—it's fine to produce no output when the codebase already has strong conventions or when review comments don't cluster into recurring themes.
89+
90+
When creating skills, place them in `.openhands/skills/{domain-name}/SKILL.md`:
91+
92+
```yaml
93+
---
94+
name: database-queries
95+
description: Database query patterns and best practices for this repository.
96+
---
97+
98+
# Database Query Guidelines
99+
100+
### Always Use Parameterized Queries
101+
[Pattern description with examples]
102+
103+
### Connection Pool Management
104+
[Pattern description with examples]
105+
```
106+
107+
Prefer skills over AGENTS.md updates, since AGENTS.md typically already contains general coding guidelines.
108+
109+
### Step 6: Create Draft PR (if applicable)
110+
111+
Use the `create_pr` tool to open a draft PR with the proposed changes. The PR description should include:
112+
- Number of PRs analyzed
113+
- Number of comments processed
114+
- Categories of patterns found
115+
- List of proposed changes (new skills and/or AGENTS.md updates)
116+
117+
## Example Output
118+
119+
### Sample Skill: API Error Handling
120+
121+
```yaml
122+
---
123+
name: api-error-handling
124+
description: API error handling patterns for this repository.
125+
---
126+
127+
# API Error Handling
128+
129+
## Always Return Structured Errors
130+
131+
❌ Avoid:
132+
```python
133+
return {"error": str(e)}
134+
```
135+
136+
✅ Prefer:
137+
```python
138+
return {
139+
"error": {
140+
"code": "VALIDATION_ERROR",
141+
"message": "Invalid input",
142+
"details": {"field": "email", "reason": "Invalid format"}
143+
}
144+
}
145+
```
146+
147+
## Log Before Returning Errors
148+
149+
```python
150+
logger.error(f"API error in {endpoint}: {e}", exc_info=True)
151+
return error_response(e)
152+
```
153+
```
154+
155+
## Defaults
156+
157+
This workflow analyzes PRs from the past 30 days by default.
158+
159+
## Best Practices
160+
161+
1. **Run periodically** - Schedule monthly or quarterly to capture evolving patterns
162+
2. **Review before merging** - Generated content is a draft; human review is essential
163+
3. **Iterate** - Refine patterns based on team feedback
164+
4. **Avoid duplication** - Check existing AGENTS.md and skills before adding
165+
5. **Cite sources** - Reference PR numbers when documenting patterns
166+
167+
## Error Handling
168+
169+
Handle these common edge cases gracefully:
170+
171+
- **Repository has few PRs**: If fewer than 10 merged PRs exist in the timeframe, inform the user that there may not be enough data to identify patterns. Proceed with analysis but note the limited sample size.
172+
- **No patterns emerge**: When comments don't cluster into recurring themes (common for well-established codebases), report this to the user and suggest either expanding the time range or that the codebase may already have strong conventions.
173+
- **Token lacks repository access**: If the GitHub API returns 403/404, explain that the token may not have access to the repository and suggest checking token permissions.
174+
- **`gh` CLI unavailable**: Fall back to direct GitHub API calls using `curl` with `$GITHUB_TOKEN`, or inform the user that `gh` needs to be installed.
175+
176+
## Limitations
177+
178+
- Only analyzes accessible repositories (requires appropriate permissions)
179+
- Cannot capture verbal feedback from pair programming or meetings
180+
- Patterns may reflect individual reviewer preferences vs. team consensus
181+
- Historical comments may reference outdated code patterns
182+
183+
## Additional Resources
184+
185+
For posting structured code reviews, see the `github-pr-review` skill.
186+
For creating new skills, see the `skill-creator` skill.

0 commit comments

Comments
 (0)