chore: add PR validation workflow and update contributing guidelines#2777
Open
fallenbagel wants to merge 1 commit intodevelopfrom
Open
chore: add PR validation workflow and update contributing guidelines#2777fallenbagel wants to merge 1 commit intodevelopfrom
fallenbagel wants to merge 1 commit intodevelopfrom
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
💤 Files with no reviewable changes (1)
✅ Files skipped from review due to trivial changes (2)
📝 WalkthroughWalkthroughAdds a new PR validation GitHub Actions workflow that validates semantic titles and PR templates (via a Node.js script), removes the old semantic-pr workflow, and updates CONTRIBUTING.md to tighten AI-disclosure and template compliance guidance. Changes
Sequence Diagram(s)sequenceDiagram
participant Contributor
participant GH_Actions as "GitHub Actions (runner)"
participant NodeScript as "bin/check-pr-template.mjs"
participant GH_API as "GitHub API (comments/labels)"
Contributor->>GH_Actions: open/reopen/edit/synchronize PR
GH_Actions->>GH_Actions: run semantic-title job
GH_Actions->>GH_Actions: run template-check job (unless synchronize)
alt PR author is bot
GH_Actions-->>Contributor: emit skip=true (skip validation)
else non-bot author
GH_Actions->>NodeScript: run check-pr-template with PR body
NodeScript-->>GH_Actions: return exit code + issues JSON
alt issues found
GH_Actions->>GH_API: upsert bot comment with "### Issues found"
GH_Actions->>GH_API: add `blocked:template` label (create if needed)
GH_Actions-->>GH_Actions: mark job failed
else no issues
GH_Actions->>GH_API: remove `blocked:template` label (if present)
GH_Actions->>GH_API: delete matching bot comment (if present)
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~30 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
fallenbagel
requested review from
a team and
Copilot
and removed request for
a team
Contributor
There was a problem hiding this comment.
Pull request overview
Adds automated PR metadata enforcement to reduce low-effort/non-template PR submissions by validating PR titles and PR-template completeness via GitHub Actions, and updates contributor guidance to set expectations around template compliance and AI disclosure.
Changes:
- Add a
PR Validationworkflow that enforces semantic PR titles and validates PR template completion (labeling/commenting on failures). - Introduce a Node-based PR body validator script (
bin/check-pr-template.mjs) used by the workflow. - Expand
CONTRIBUTING.mdguidance around PR template compliance and AI disclosure expectations; remove the old standalonesemantic-prworkflow.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
bin/check-pr-template.mjs |
New script to parse PR body and report missing/placeholder sections and unchecked required checklist items. |
CONTRIBUTING.md |
Clarifies AI assistance disclosure requirements and sets stricter expectations for filling out the PR template. |
.github/workflows/semantic-pr.yml |
Removes the legacy standalone semantic PR title workflow (replaced by consolidated validation workflow). |
.github/workflows/pr-validation.yml |
New consolidated workflow: semantic title check + PR template validation with automatic labeling/commenting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
.github/workflows/pr-validation.yml (1)
90-99: Mark the bot comment explicitly before updating or deleting it.Matching any bot comment that contains
### Issues found:is too broad. This can overwrite or delete an unrelated automation comment. Add a workflow-specific marker tocommentBodyand match on that marker instead.Suggested fix
+ const COMMENT_MARKER = '<!-- pr-template-validation -->'; const issueList = issues.map(i => `- ${i}`).join('\n'); const commentBody = [ + COMMENT_MARKER, `Hey @${author}, thanks for submitting this PR! However, it looks like the PR template hasn't been fully filled out.\n`, `### Issues found:\n`, issueList, @@ const botComment = comments.data.find( - c => c.user.type === 'Bot' && c.body.includes('### Issues found:') + c => c.user.type === 'Bot' && c.body.includes(COMMENT_MARKER) ); @@ const botComment = comments.data.find( - c => c.user.type === 'Bot' && c.body.includes('### Issues found:') + c => c.user.type === 'Bot' && c.body.includes(COMMENT_MARKER) );Also applies to: 108-110, 184-185
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/pr-validation.yml around lines 90 - 99, The bot's commentBody (variable commentBody) currently uses a generic marker `### Issues found:` which is too broad and risks matching unrelated automation comments; update the commentBody to include a unique workflow-specific marker (e.g., a short, unique token or HTML comment like `<!-- pr-validation-bot: v1 -->`) and then change any comment-matching/updating logic to look for that exact marker instead of `### Issues found:`; apply the same change to the other commentBody instances referenced in the review so every workflow-specific bot comment has the unique marker used for safe updates/deletes.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/pr-validation.yml:
- Line 86: The workflow is using a generic LABEL constant ('blocked') that this
job later unconditionally removes; change LABEL to a workflow-owned name (e.g.,
'pr-template-blocked' or similar) and update all uses (the const LABEL and every
place that adds/removes it) so the job only touches that workflow-specific
label; additionally, when removing the label, guard the removal so it only
removes labels the workflow actually added (for example by checking the label
name matches the workflow-owned name or verifying the label was created/added by
the bot/this workflow) rather than unconditionally removing a generic 'blocked'
label.
In `@bin/check-pr-template.mjs`:
- Around line 19-22: The maintainer exemption implemented by MAINTAINER_ROLES
and isMaintainer lets OWNER/MEMBER/COLLABORATOR bypass the AI-disclosure
checkbox, which conflicts with CONTRIBUTING.md; either remove the exemption
logic or make it explicit in the guide. Update the code around MAINTAINER_ROLES
/ isMaintainer (and any checks that use isMaintainer) to require the AI
disclosure for all authors by removing/ignoring this maintainer whitelist, or if
you must keep it, add a clear comment and link to the documented exemption and
adjust CONTRIBUTING.md to describe the exception; ensure
process.env.AUTHOR_ASSOCIATION is still read but no longer used to skip
AI-disclosure validation (or document the exemption in the guide).
- Around line 24-29: The replacement only strips triple-dash comments (`<!---
... --->`) so template placeholders like `<!-- ... -->` remain and make empty
sections appear non-empty; update the comment-stripping regex used when building
descriptionContent (and the equivalent replacement at the other location
handling "How Has This Been Tested?") to match standard HTML comments (use a
pattern that matches `<!-- ... -->`, e.g., `<!--[\s\S]*?-->` with the same
flags) and ensure both occurrences (the logic around
descriptionMatch/descriptionContent and the similar testing-section variable)
use that corrected regex before trimming and checking emptiness.
In `@CONTRIBUTING.md`:
- Around line 66-69: Fix the typo in the maintainer-review paragraph by
replacing the garbled phrase "isn not a maintainers job" with the correct
wording, e.g., "isn't a maintainer's job" (or "is not a maintainer's job") so
the sentence reads clearly; update the string containing that sentence in the
CONTRIBUTING.md paragraph where the phrase appears.
- Around line 41-48: The blockquote examples are split by blank lines
(triggering MD028); remove the empty lines inside each example so each
blockquote is a single contiguous quoted paragraph—specifically make the three
example lines starting with "> **AI Disclosure:**" contiguous (no blank lines
between the '>' lines) so each example is one continuous blockquote.
---
Nitpick comments:
In @.github/workflows/pr-validation.yml:
- Around line 90-99: The bot's commentBody (variable commentBody) currently uses
a generic marker `### Issues found:` which is too broad and risks matching
unrelated automation comments; update the commentBody to include a unique
workflow-specific marker (e.g., a short, unique token or HTML comment like `<!--
pr-validation-bot: v1 -->`) and then change any comment-matching/updating logic
to look for that exact marker instead of `### Issues found:`; apply the same
change to the other commentBody instances referenced in the review so every
workflow-specific bot comment has the unique marker used for safe
updates/deletes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 12935931-2574-4035-a8af-5ba07cfafc05
📒 Files selected for processing (4)
.github/workflows/pr-validation.yml.github/workflows/semantic-pr.ymlCONTRIBUTING.mdbin/check-pr-template.mjs
💤 Files with no reviewable changes (1)
- .github/workflows/semantic-pr.yml
fallenbagel
force-pushed
the
fallenbagel/chore/pr-template-check-with-updated-contributing-guides
branch
from
ab05dc4 to
c20b8f9
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
CONTRIBUTING.md (1)
68-68:⚠️ Potential issue | 🟡 MinorFix possessive typo in maintainer-review sentence.
Line 68 should use possessive form:
maintainer's(ormaintainers', if plural possessive is intended).Suggested fix
-is not a maintainers job to review a PR so broken that it requires +is not a maintainer's job to review a PR so broken that it requires🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` at line 68, Replace the non-possessive phrase "is not a maintainers job to review a PR so broken that it requires" with the correct possessive form; change to either "is not a maintainer's job to review a PR so broken that it requires" (singular) or "is not a maintainers' job to review a PR so broken that it requires" (plural) depending on the intended meaning, updating the sentence containing the exact phrase "is not a maintainers job to review a PR so broken that it requires".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@bin/check-pr-template.mjs`:
- Around line 29-30: The current .replace call that removes the placeholder ("-
Fixes `#XXXX`") is too strict; update both .replace(...) usages (the calls that
currently use /- Fixes `#XXXX/g`) to use a case-insensitive pattern that matches
variants with optional surrounding backticks/quotes/dashes and arbitrary
whitespace (e.g., accepts "Fixes `#XXXX`", "`fixes `#XXXX``", "- fixes `#XXXX`",
etc.), then trim as before; apply the same change to the second occurrence
around lines 77-79 so all placeholder variants are removed reliably.
---
Duplicate comments:
In `@CONTRIBUTING.md`:
- Line 68: Replace the non-possessive phrase "is not a maintainers job to review
a PR so broken that it requires" with the correct possessive form; change to
either "is not a maintainer's job to review a PR so broken that it requires"
(singular) or "is not a maintainers' job to review a PR so broken that it
requires" (plural) depending on the intended meaning, updating the sentence
containing the exact phrase "is not a maintainers job to review a PR so broken
that it requires".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 5d511490-6345-4403-832d-d8190460ade7
📒 Files selected for processing (4)
.github/workflows/pr-validation.yml.github/workflows/semantic-pr.ymlCONTRIBUTING.mdbin/check-pr-template.mjs
💤 Files with no reviewable changes (1)
- .github/workflows/semantic-pr.yml
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/pr-validation.yml
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 4 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…for AI assistance
fallenbagel
force-pushed
the
fallenbagel/chore/pr-template-check-with-updated-contributing-guides
branch
from
c20b8f9 to
cad6e3a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We've been getting an increasing number of PRs that completely ignore the PR template with no description, no testing details, unchecked checklists, and AI-generated descriptions that replace our template with their own format. This adds an automated PR template validation check that labels non-compliant PRs as
blockedand comments with what's missing. It also updates the contributing guide to make it explicit that ignoring the PR template may result in closure without review.How Has This Been Tested?
Screenshots / Logs (if applicable)
Checklist:
pnpm buildpnpm i18n:extractSummary by CodeRabbit
Documentation
Chores