Breaking change doc tool #120987
Breaking change doc tool #120987
Conversation
github-actions
bot
added
the
needs-area-label
There was a problem hiding this comment.
Pull Request Overview
This PR introduces an automated workflow for creating breaking change documentation. When a PR is labeled needs-breaking-change-doc-created, a PowerShell script analyzes the PR details and uses an LLM to generate documentation content. The generated content is then encoded into a clickable GitHub issue creation link, which is posted as a comment on the PR.
Key Changes
- PowerShell script that collects PR data, analyzes it with LLM, and generates breaking change documentation
- GitHub Actions workflow that triggers on label addition to run the automation
- Configuration file for customizing LLM provider, repository settings, and other parameters
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| eng/breakingChanges/config.ps1 | Configuration file defining LLM settings, GitHub repository paths, issue template settings, and rate limiting parameters |
| eng/breakingChanges/breaking-change-doc.ps1 | Main PowerShell script implementing the breaking change documentation workflow with data collection, LLM analysis, and issue generation |
| eng/breakingChanges/README.md | Documentation explaining the automation tool's features, setup instructions, usage examples, and configuration options |
| .github/workflows/breaking-change-doc.yml | GitHub Actions workflow that triggers the script when a PR receives the needs-breaking-change-doc-created label |
|
This is testing the workflow - #120988 Looks like it's failing to fetch the issues from dotnet/docs. I'll see if I can get it issue read permission to the docs repo. |
|
Now seeing Need to find out how to use models from a workflow. |
rzikm
left a comment
rzikm
left a comment
There was a problem hiding this comment.
Looks great, I look forward to using this once the technical hurdles get ironed out.
|
This works file even without the workflow, so it can be used by directly running the script after installing the github CLI. Worse case we'll just use that and occasionally run it. The summary of the problems with the workflow is that our workflows in the |
ericstj
added
area-Meta
and removed
needs-area-label
|
Tagging subscribers to this area: @dotnet/area-meta |
|
I don't expect this to work, but also trying copilot CLI provider. |
|
@jeffhandley @dotnet/runtime-infrastructure can I get a review? If folks are worried about the workflow + secret used here I can remove that part and just keep the scripts, but I think the automation adds value. |
The workflow will now run when a PR is closed, or a label is appled. In both cases it only proceeds if the PR has both been merged and has the breaking change doc label.
|
it looks like the changes to URL escaping broke the formatting of newlines. I'll need to see when that happened and undo it. |
|
I actually broke it when refactoring how we call the gh CLI. I also noticed some concern with passing prompts on the commandline. We should avoid this - instead passing a prompt as a file. |
| generate-breaking-change-doc: | ||
| if: | | ||
| github.repository_owner == 'dotnet' && ( | ||
| (github.event_name == 'pull_request_target' && github.event.action == 'closed' && github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'needs-breaking-change-doc-created')) || |
There was a problem hiding this comment.
Very nice usage of github.event.pull_request.labels.*.name 💯
|
|
||
| EXAMPLES: | ||
| .\breaking-change-doc.ps1 -PrNumber 114929 # Process specific PR | ||
| .\breaking-change-doc.ps1 -Query "repo:dotnet/runtime state:closed label:needs-breaking-change-doc-created is:merged merged:>2024-09-16 -milestone:11.0.0" |
There was a problem hiding this comment.
When running in Query mode, the API call includes --repo $Config.SourceRepo. Is the repo:dotnet/runtime required to be part of the query as well?
| $closingIssuesInfo = if ($pr.ClosingIssues -and $pr.ClosingIssues.Count -gt 0) { | ||
| $issuesList = $pr.ClosingIssues | ForEach-Object { | ||
| $issueComments = if ($_.Comments -and $_.Comments.Count -gt 0) { | ||
| "Comments: $($_.Comments.Count) comments available" | ||
| } else { | ||
| "No comments" | ||
| } |
There was a problem hiding this comment.
We recently hit upon a gap in our processes where we can have both issues and PRs labeled as
breaking-change
I don't think there's anything to do in this PR for that, but seeing the traversal of ClosingIssues here has me wondering if we should refine our approach for labeling--perhaps to automatically label merged PRs as breaking changes if a breaking-change issue was closed by the PR.
This adds a workflow to run a script when a PR is labeled as
needs-breaking-change-doc-created,The script collects details about the PR, the breaking change issue template, and sample issues then asks an LLM to generate an issue based on that.
The issue content is then encoded into a clickable link which is then added as a comment to the PR.
I'm doing this from a dotnet branch so that I can hopefully create a PR off of it to demonstrate the workflow in action.