Skip to content

Commit 4b8a655

Browse files
ericstjericstj
authored
Breaking change doc tool (#120987)
This adds a workflow to run a script when a PR is labeled as `needs-breaking-change-doc-created`, The script collects details about the PR, the breaking change issue template, and sample issues then asks an LLM to generate an issue based on that. The issue content is then encoded into a clickable link which is then added as a comment to the PR. I'm doing this from a dotnet branch so that I can hopefully create a PR off of it to demonstrate the workflow in action.
1 parent 20901cb commit 4b8a655

File tree

4 files changed

+1508
-0
lines changed

4 files changed

+1508
-0
lines changed

‎.github/workflows/breaking-change-doc.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# This workflow generates breaking change documentation for merged pull requests.
2+
# It runs automatically when a PR with the 'needs-breaking-change-doc-created' label is merged,
3+
# or when that label is added to an already merged PR.
4+
# It can be manually triggered to generate documentation for any specific PR.
5+
#
6+
# The workflow uses GitHub Models AI to analyze the PR changes and create appropriate
7+
# breaking change documentation that gets posted as a PR comment as a clickable link
8+
# to open an issue in the dotnet/docs repository.
9+
name: Breaking Change Documentation
10+
11+
on:
12+
pull_request_target:
13+
types: [closed, labeled]
14+
workflow_dispatch:
15+
inputs:
16+
pr_number:
17+
description: "Pull Request Number"
18+
required: true
19+
type: number
20+
21+
permissions:
22+
contents: read
23+
pull-requests: write
24+
models: read
25+
26+
jobs:
27+
generate-breaking-change-doc:
28+
if: |
29+
github.repository_owner == 'dotnet' && (
30+
(github.event_name == 'pull_request_target' && github.event.action == 'closed' && github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'needs-breaking-change-doc-created')) ||
31+
(github.event_name == 'pull_request_target' && github.event.action == 'labeled' && github.event.pull_request.merged == true && github.event.label.name == 'needs-breaking-change-doc-created') ||
32+
github.event_name == 'workflow_dispatch'
33+
)
34+
runs-on: ubuntu-latest
35+
36+
steps:
37+
- name: Checkout repository
38+
uses: actions/checkout@v4
39+
with:
40+
fetch-depth: 0 # Need full history for version detection
41+
42+
- name: Verify PowerShell
43+
run: |
44+
pwsh --version
45+
46+
- name: Verify GitHub CLI
47+
run: |
48+
gh --version
49+
50+
- name: Install GitHub Models extension
51+
run: |
52+
gh extension install github/gh-models --force
53+
env:
54+
GH_TOKEN: ${{ github.token }}
55+
56+
- name: Fetch latest tags
57+
run: |
58+
git fetch --tags --force
59+
60+
- name: Run breaking change documentation script
61+
shell: pwsh
62+
working-directory: eng/breakingChanges
63+
run: ./breaking-change-doc.ps1 -PrNumber ${{ inputs.pr_number || github.event.pull_request.number }} -Comment
64+
env:
65+
GH_TOKEN: ${{ github.token }}
66+
GITHUB_MODELS_API_KEY: ${{ secrets.MODELS_TOKEN }}
67+
68+
- name: Upload artifacts
69+
uses: actions/upload-artifact@v4
70+
with:
71+
name: breaking-change-doc-artifacts-${{ inputs.pr_number || github.event.pull_request.number }}
72+
path: artifacts/docs/breakingChanges/
73+
retention-days: 7

‎eng/breakingChanges/README.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# Breaking Change Documentation Automation
2+
3+
This script automates the creation of high-quality breaking change documentation for .NET runtime PRs using AI-powered analysis.
4+
5+
## Key Features
6+
7+
- **GitHub Models Integration**: Uses GitHub's AI models (no API keys required) with fallback to other providers
8+
- **Dynamic Template Fetching**: Automatically fetches the latest breaking change issue template from dotnet/docs
9+
- **Example-Based Learning**: Analyzes recent breaking change issues to improve content quality
10+
- **Version Detection**: Analyzes GitHub tags to determine accurate .NET version information for proper milestone assignment
11+
- **Flexible Workflow**: Multiple execution modes (CollectOnly, Comment, CreateIssues) with analysis-only default
12+
- **Comprehensive Data Collection**: Gathers PR details, related issues, merge commits, review comments, and closing issues
13+
- **Area Label Detection**: Automatically detects feature areas from GitHub labels (area-*) with file path fallback
14+
- **Individual File Output**: Creates separate JSON files per PR for easy examination
15+
16+
## Quick Setup
17+
18+
1. **Install Prerequisites:**
19+
- GitHub CLI: `gh auth login`
20+
- Choose LLM provider:
21+
- **GitHub Models** (recommended): `gh extension install github/gh-models`
22+
- **OpenAI**: Set `$env:OPENAI_API_KEY = "your-key"`
23+
- **Others**: See configuration section below
24+
25+
2. **Configure:**
26+
```powershell
27+
# Edit config.ps1 to set:
28+
# - LlmProvider = "github-models" (or other provider)
29+
```
30+
31+
3. **Run the workflow:**
32+
```powershell
33+
.\breaking-change-doc.ps1 -Help
34+
```
35+
36+
4. **Choose your workflow:**
37+
```powershell
38+
# Default: Analysis only (generates drafts without making GitHub changes)
39+
.\breaking-change-doc.ps1 -PrNumber 123456
40+
41+
# Add comments with create issue links
42+
.\breaking-change-doc.ps1 -PrNumber 123456 -Comment
43+
44+
# Create issues directly
45+
.\breaking-change-doc.ps1 -PrNumber 123456 -CreateIssues
46+
47+
# Just collect data
48+
.\breaking-change-doc.ps1 -PrNumber 123456 -CollectOnly
49+
```
50+
51+
## Commands
52+
53+
```powershell
54+
# Help (shows all parameters and examples)
55+
.\breaking-change-doc.ps1 -Help
56+
57+
# Default workflow (analysis only - generates drafts)
58+
.\breaking-change-doc.ps1 -PrNumber 123456
59+
60+
# Add comments with issue creation links
61+
.\breaking-change-doc.ps1 -PrNumber 123456 -Comment
62+
63+
# Create issues directly
64+
.\breaking-change-doc.ps1 -PrNumber 123456 -CreateIssues
65+
66+
# Data collection only
67+
.\breaking-change-doc.ps1 -PrNumber 123456 -CollectOnly
68+
69+
# Query multiple PRs
70+
.\breaking-change-doc.ps1 -Query "repo:dotnet/runtime state:closed label:needs-breaking-change-doc-created is:merged"
71+
72+
# Clean previous data
73+
.\breaking-change-doc.ps1 -Clean
74+
75+
# Clean and process
76+
.\breaking-change-doc.ps1 -Clean -PrNumber 123456
77+
```
78+
79+
## Configuration
80+
81+
Edit `config.ps1` to customize:
82+
- **LLM provider**: GitHub Models, OpenAI, Anthropic, Azure OpenAI
83+
- **Search parameters**: Date ranges, labels, excluded milestones
84+
- **Output settings**: Labels, assignees, notification emails
85+
86+
## LLM Providers
87+
88+
**GitHub Models** (recommended - no API key needed):
89+
```powershell
90+
gh extension install github/gh-models
91+
# Set provider in config.ps1: LlmProvider = "github-models"
92+
```
93+
94+
**OpenAI**:
95+
```powershell
96+
$env:OPENAI_API_KEY = "your-key"
97+
# Set provider in config.ps1: LlmProvider = "openai"
98+
```
99+
100+
**Anthropic Claude**:
101+
```powershell
102+
$env:ANTHROPIC_API_KEY = "your-key"
103+
# Set provider in config.ps1: LlmProvider = "anthropic"
104+
```
105+
106+
**Azure OpenAI**:
107+
```powershell
108+
$env:AZURE_OPENAI_API_KEY = "your-key"
109+
# Configure endpoint in config.ps1: LlmProvider = "azure-openai"
110+
```
111+
112+
## Output
113+
114+
- **Data Collection**: `(repoRoot)\artifacts\docs\breakingChanges\data\summary_report.md`, `(repoRoot)\artifacts\docs\breakingChanges\data\pr_*.json`
115+
- **Issue Drafts**: `(repoRoot)\artifacts\docs\breakingChanges\issue-drafts\*.md`
116+
- **Comment Drafts**: `(repoRoot)\artifacts\docs\breakingChanges\comment-drafts\*.md`
117+
- **GitHub Issues**: Created automatically when using -CreateIssues
118+
- **GitHub Comments**: Added to PRs when using -Comment
119+
120+
## Workflow Steps
121+
122+
1. **Fetch PRs** - Downloads PR data from dotnet/runtime with comprehensive details
123+
2. **Version Detection** - Analyzes GitHub tags to determine accurate .NET version information
124+
3. **Template & Examples** - Fetches latest issue template and analyzes recent breaking change issues
125+
3. **AI Analysis** - Generates high-quality breaking change documentation using AI
126+
4. **Output Generation** - Creates issue drafts and comment drafts for review
127+
5. **Optional Actions** - Adds comments with issue creation links (-Comment) or creates issues directly (-CreateIssues)
128+
129+
## Version Detection
130+
131+
The script automatically determines accurate .NET version information using the local git repository:
132+
- **Fast and reliable**: Uses `git describe` commands on the repository
133+
- **No API rate limits**: Avoids GitHub API calls for version detection
134+
- **Accurate timing**: Analyzes actual commit ancestry and tag relationships
135+
- **Merge commit analysis**: For merged PRs, finds the exact merge commit and determines version context
136+
- **Branch-aware**: For unmerged PRs, uses target branch information
137+
138+
## Manual Review
139+
140+
AI generates 90%+ ready documentation, but review for:
141+
- Technical accuracy
142+
- API completeness
143+
- Edge cases
144+
145+
## Cleanup
146+
147+
Between runs:
148+
```powershell
149+
.\breaking-change-doc.ps1 -Clean
150+
```
151+
152+
## Parameters
153+
154+
| Parameter | Description | Example |
155+
|-----------|-------------|---------|
156+
| `-Help` | Show help and parameter information | `.\breaking-change-doc.ps1 -Help` |
157+
| `-PrNumber` | Process a specific PR number | `.\breaking-change-doc.ps1 -PrNumber 123456` |
158+
| `-Query` | GitHub search query for multiple PRs | `.\breaking-change-doc.ps1 -Query "repo:dotnet/runtime state:closed label:needs-breaking-change-doc-created is:merged"` |
159+
| `-CollectOnly` | Only collect PR data, don't generate documentation | `.\breaking-change-doc.ps1 -PrNumber 123456 -CollectOnly` |
160+
| `-Comment` | Add comments to PRs with issue creation links | `.\breaking-change-doc.ps1 -PrNumber 123456 -Comment` |
161+
| `-CreateIssues` | Create GitHub issues directly | `.\breaking-change-doc.ps1 -PrNumber 123456 -CreateIssues` |
162+
| `-Clean` | Clean previous data before starting | `.\breaking-change-doc.ps1 -Clean` |
163+
164+
**Note**: Either `-PrNumber` or `-Query` must be specified (unless using `-Clean` or `-Help` alone).
165+
166+
## Troubleshooting
167+
168+
**GitHub CLI**: `gh auth status` and `gh auth login`
169+
**API Keys**: Verify environment variables are set for non-GitHub Models providers
170+
**Rate Limits**: Script includes delays between API calls
171+
**Git Operations**: Ensure git is in PATH and repository is up to date (`git fetch --tags`)
172+
**Parameter Issues**: Use `-Help` to see current parameter list and examples

0 commit comments

Comments
 (0)