fix: Add mandatory CI check verification before claiming PR review complete

[rjmurillo-bot]

Problem

PR comment responder claimed PR #199 was "MERGEABLE" based on incomplete verification. The agent resolved all review threads and checked mergeable status, but failed to verify CI checks were passing.

Evidence: Run 20487393463 failed with MUST: HANDOFF.md Updated: FAIL in session 56, but the agent claimed completion without checking CI.

Root Cause Analysis

What Was Checked

1. ✅ Resolved all 11 review threads via GraphQL mutations
2. ✅ Verified 0 unresolved threads via GraphQL query
3. ✅ Ran gh pr view 199 --json state,mergeable,reviewDecision → {"mergeable":"MERGEABLE",...}

What Was Missed

❌ Did NOT run gh pr checks 199 to verify CI status

The mergeable: "MERGEABLE" field only indicates:

• No merge conflicts
• Branch is compatible with base

It does NOT indicate:

• CI checks passing
• Required status checks satisfied

The Flawed Logic

Agent checked: mergeable == "MERGEABLE" + reviewThreads all resolved
Agent concluded: PR is ready to merge

Correct check should be:
- mergeable == "MERGEABLE" (no conflicts)
- reviewThreads all resolved
- gh pr checks all SUCCESS  ← MISSING
- no new comments after wait ← MISSING

Completion Criteria Gap

The .claude/commands/pr-review.md template correctly lists:

Criterion	Verification	Required
All comments resolved	Each comment has [COMPLETE] or [WONTFIX] status	Yes
No new comments	Re-check after 45s wait returned 0 new	Yes
CI checks pass	gh pr checks all green	Yes
No unresolved threads	GraphQL query for unresolved reviewThreads	Yes
Commits pushed	git status shows "up to date with origin"	Yes

The agent verified criteria 1, 4, and 5 but skipped criteria 2 and 3.

Proposed Fix

1. Update pr-comment-responder agent

Add mandatory CI verification step before Phase 8 completion:

# After all other verification, BEFORE claiming completion
FAILED_CHECKS=$(gh pr checks $PR_NUMBER --json name,state,conclusion | \
  jq '[.[] | select(.conclusion != "success" and .conclusion != "skipped" and .conclusion != null)]')

if [ "$(echo "$FAILED_CHECKS" | jq 'length')" -gt 0 ]; then
  echo "[BLOCKED] CI checks not passing:"
  echo "$FAILED_CHECKS" | jq -r '.[].name'
  # Do NOT claim completion
fi

2. Update /pr-review skill

Add verification step after pushing commits:

# Wait for CI to start (avoid race condition)
sleep 10

# Check CI status
gh pr checks $PR_NUMBER --watch --fail-fast || {
  echo "[BLOCKED] CI checks failed. Cannot claim completion."
  exit 1
}

3. Create Serena memory skill

Store as skill-pr-review-007-ci-verification.md:

• Pattern: Always verify CI before claiming completion
• Trigger: After resolving review threads and pushing commits
• Verification: gh pr checks returns all success/skipped

Acceptance Criteria

• PR comment responder runs gh pr checks before claiming completion
• /pr-review skill waits for CI after pushing commits
• Summary output includes CI status for each check
• If CI fails, agent does NOT claim PR is mergeable
• Skill memory created and indexed

Related

• Issue feat: Add PR merge state verification to prevent wasted review effort #321: PR merge state verification (pre-review) - different phase
• Issue [EPIC] Pre-PR Validation System: Prevent premature PR opening across all agents #265: Pre-PR validation (before opening) - different phase
• PR feat(agents): add mandatory memory phases to pr-comment-responder #199: Current PR where this failure was discovered

[latta-ai]

A new task Issue: fix: Add mandatory CI check verification before claiming PR review complete has been created with an AI Solve step.
AI Solve is now analyzing the reported problem.

[github-actions]

AI Triage Summary

Note

This issue has been automatically triaged by AI agents

What is AI Triage?

This issue was analyzed by AI agents:

• Analyst Agent: Categorizes the issue and suggests appropriate labels
• Roadmap Agent: Aligns the issue with project milestones and priorities
• Explainer Agent (if escalated): Generates comprehensive PRD

Triage Results

Property	Value
Category	bug
Labels	["bug","agent-qa","area-workflows"]
Priority	P1
Milestone	v1.1

Categorization Analysis

```json
{
  "labels": ["bug", "agent-qa", "area-workflows"],
  "category": "bug",
  "confidence": 0.92,
  "reasoning": "Issue describes a verification failure in the PR comment responder agent that missed CI check validation before claiming completion"
}

</details>

<details>
<summary>Roadmap Alignment</summary>

```json
Based on my review of the roadmap and product context, this issue aligns with the agent system quality and skills management area. The issue is well-documented with clear root cause analysis and acceptance criteria, making it a straightforward bug fix with process improvement.

```json
{
  "milestone": "v1.1",
  "priority": "P1",
  "epic_alignment": "",
  "confidence": 0.85,
  "reasoning": "Bug in pr-comment-responder agent with clear fix path; affects release quality but does not block core functionality",
  "escalate_to_prd": false,
  "escalation_criteria": [],
  "complexity_score": 0
}

</details>

---

<sub>Powered by [AI Issue Triage](https://github.com/rjmurillo/ai-agents) - [View Workflow](https://github.com/rjmurillo/ai-agents/actions/runs/20487595340)</sub>

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• fix(ci): Required check 'Validate Memory Files' references non-existent workflow #341
• feat: Add PR merge state verification to prevent wasted review effort #321
• [P1] Implement Technical Guardrails for Autonomous Agent Execution #230
• agent/implementer: Add pre-PR validation checklist to prevent premature PR opening #257
• agent/qa: Add mandatory pre-PR quality gate enforcement #258

🔗 Related PRs

rjmurillo/moq.analyzers#552 - Update copilot instructions for PR checks and CI [merged]
#100 - fix(ci): ensure Pester Tests workflow satisfies required checks for all PRs [merged]
#141 - feat(skills): document dorny/paths-filter checkout requirement [merged]
#201 - feat(skills): import CodeRabbit AI learnings as validated skills [merged]
#229 - fix: address PR #226 review comments and labeler workflow issues [merged]
#296 - fix(workflow): ensure copilot synthesis posts comment on successful AI output [merged]
#298 - fix(ci): ensure Copilot Workspace Setup exits with success [merged]

👤 Suggested Assignees

• rjmurillo-bot
• rjmurillo

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[latta-ai]

AI Solve completed successfully.

Summary
A skill for mandatory CI check verification was added to the PR review process, with corresponding updates to the PR comment responder agent and skills index.

Merge request
#371

The AI Solve step is now marked as complete.

[coderabbitai]

Coding Plan

Approach

Add mandatory CI check verification to the PR comment responder workflow to prevent claiming PR completion when CI checks are failing. The agent currently verifies review threads are resolved and mergeable status is clean, but skips CI verification despite existing Phase 8.4 documentation requiring it.

Observations

The project uses a template-based agent generation system where templates/agents/*.shared.md files are transformed by build/Generate-Agents.ps1 into platform-specific outputs. The pr-comment-responder agent coordinates PR review responses and delegates to the orchestrator for implementation. Phase 8.4 already exists in documentation with CI verification logic, but the agent bypassed it in practice. Validation occurs via scripts/Validate-SessionEnd.ps1 which checks session protocol compliance but does not verify task-specific completion criteria like CI status.

Assumptions

Assumption 1: How should CI verification be enforced when Phase 8.4 already documents it as MANDATORY?

Options Considered:

• Add stronger documentation language only (warning markers, examples)
• Add validation script enforcement that checks for CI verification evidence in session logs
• Create structured completion artifacts that validation scripts can verify

Chosen Option: Evidence-based enforcement through validation scripts combined with clearer phase structure

Rationale: This balances enforcement (catching the gap) with alignment to existing patterns (extending validation scripts rather than creating new artifact systems). Developers already understand the validation script pattern, and extending it to check for CI evidence feels natural. This approach provides actual enforcement without requiring major architectural changes.

Assumption 2: Should CI verification happen before or after review thread resolution?

Options Considered:

• After thread resolution (current Phase 8.4 position) - verify everything at the end
• Before claiming any completion - make CI a blocking gate earlier in Phase 8
• Parallel check - verify CI while resolving threads

Chosen Option: Restructure Phase 8 to check CI status early (Phase 8.1) before proceeding with thread resolution claims

Rationale: This makes CI a blocking gate that prevents wasted work if CI is already failing, and makes the failure visible immediately rather than at the end. Aligns with fail-fast principles common in development workflows.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Restructure Phase 8 CI Verification Flow

Reorganize Phase 8 in the pr-comment-responder template to make CI verification a prominent early gate rather than a late-stage check. This prevents agents from resolving threads and claiming partial completion when CI is already failing.

Task 1: Move CI Verification to Phase 8.1

Modify templates/agents/pr-comment-responder.shared.md Phase 8 structure to check CI status first.

• Rename current Phase 8.1 to Phase 8.2 (bump all existing sub-phases down)
• Create new Phase 8.1: "CI Status Verification" as the first mandatory gate
• Add BLOCKING marker with explicit failure instructions: "Do NOT proceed with review thread resolution if CI is failing"
• Include gh pr checks [number] --json name,state,conclusion command with output expectations
• Document that skipped checks count as passing (conclusion == "skipped" is acceptable)
• Add evidence requirement: agent must output CI check results to session log showing all checks are success/skipped

Task 2: Add CI Evidence Output Format

Define structured output format for CI verification results that validation can check.

• Add section in Phase 8.1 specifying required output format: "CI Status: [passing|failing] - [N] checks successful, [M] checks skipped, [P] checks failed"
• Include instruction to list any failing check names explicitly
• Add instruction to write CI status to task list with TodoWrite showing check counts
• This makes CI status visible in session artifacts that validation scripts can parse

Task 3: Update Phase 8.5 Completion Criteria

Strengthen completion criteria language to reference CI verification evidence.

• Update Phase 8.5 criteria table to add "CI verification evidence" column showing expected session log output
• Change "CI checks pass" criterion to reference Phase 8.1 output requirement
• Add note that claiming completion without Phase 8.1 evidence is a protocol violation

🤖 Prompt for AI agents

Restructure Phase 8 in the pr-comment-responder template to make CI verification
the first blocking gate.

In `templates/agents/pr-comment-responder.shared.md`:
- Renumber existing Phase 8.1 to Phase 8.2 (bump all sub-phases down)
- Create new Phase 8.1 "CI Status Verification" as first mandatory gate
- Add BLOCKING marker with instruction to not proceed if CI is failing
- Include `gh pr checks [number] --json name,state,conclusion` command
- Document that skipped checks count as passing
- Add required output format: "CI Status: [passing|failing] - [N] checks
successful, [M] checks skipped, [P] checks failed"
- Include instruction to list failing check names explicitly
- Add instruction to write CI status to task list with TodoWrite
- Update Phase 8.5 completion criteria to add CI verification evidence column
- Change "CI checks pass" criterion to reference Phase 8.1 output
- Add note that completion without Phase 8.1 evidence is a protocol violation

Phase 2: Add Validation Script Enforcement

Extend session end validation to verify CI check evidence exists in session logs, preventing completion claims when CI verification is missing.

Task 1: Add CI Evidence Check to Validate-SessionEnd.ps1

Create new validation function that checks for CI verification in PR review sessions.

• Add Test-PRCIVerification function in scripts/Validate-SessionEnd.ps1 after existing test functions
• Function should detect PR review sessions by checking if session log references pr-comment-responder agent
• Search session log for "CI Status:" evidence pattern added in Phase 1.2
• Parse the status line to verify it shows passing/skipped checks (not failing)
• Return validation result with descriptive error message if evidence missing or shows failures

Task 2: Integrate CI Check into Session Validation Flow

Wire the new CI verification check into the main validation pipeline.

• Add CI verification test to the main validation sequence in Validate-SessionEnd.ps1 where other MUST requirements are checked
• Only run this check for sessions that involve pr-comment-responder agent (use agent detection logic)
• Make it a MUST requirement (failing this check should block session end)
• Add clear error message: "PR review session missing CI verification evidence in Phase 8.1"

Task 3: Update Validation Documentation

Document the new CI verification requirement in validation-related documentation.

• Update .agents/SESSION-END-PROMPT.md to mention CI verification as a mandatory gate for PR review sessions
• Add example of the expected CI status evidence format
• Reference Phase 8.1 of pr-comment-responder as the source of this requirement

🤖 Prompt for AI agents

Add validation script enforcement to verify CI check evidence in PR review
sessions.

In `scripts/Validate-SessionEnd.ps1`:
- Create `Test-PRCIVerification` function after existing test functions
- Detect PR review sessions by checking for pr-comment-responder agent reference
- Search session log for "CI Status:" evidence pattern
- Parse status line to verify passing/skipped checks (not failing)
- Return validation result with error if evidence missing or shows failures
- Integrate CI verification test into main validation sequence with other MUST
requirements
- Only run for pr-comment-responder sessions
- Make it a MUST requirement that blocks session end
- Add error message: "PR review session missing CI verification evidence in
Phase 8.1"

In `.agents/SESSION-END-PROMPT.md`:
- Add CI verification as mandatory gate for PR review sessions
- Include example of expected CI status evidence format
- Reference Phase 8.1 of pr-comment-responder

Phase 3: Create Learning and Discovery Artifacts

Build Serena memory skill and update discovery indexes to help future agent runs find and apply the CI verification pattern.

Task 1: Create Serena Memory Skill for CI Verification

Add new skill memory documenting the CI verification requirement as a learned pattern.

• Create .serena/memories/skill-pr-007-ci-verification-required.md following existing skill memory format
• Document the pattern: "Before claiming PR review completion, verify CI checks via gh pr checks and output evidence to session log"
• Include context explaining why this matters (mergeable != CI passing)
• Add evidence section showing the Phase 8.1 verification requirement
• Set high atomicity rating (90%+) since this is a mandatory requirement with clear verification
• Include impact rating showing this prevents false completion claims

Task 2: Update PR Review Skills Index

Add the new memory to the existing PR review skills index for discoverability.

• Update .serena/memories/skills-pr-review-index.md to include entry for skill-pr-007
• Add keyword mapping: "CI checks verification mergeable status completion Phase 8.1"
• Ensure the index entry groups it with other completion verification skills (skill-pr-003)

Task 3: Add CI Status to Command Summary Output

Make CI verification visible in the pr-review command's final summary output.

• Update .claude/commands/pr-review.md Step 6 (Generate Summary) to include CI check status per PR
• Add table column or summary line showing CI status for each reviewed PR
• Use the evidence format from Phase 8.1 so developers can see at a glance which PRs have passing CI
• This makes missing CI verification immediately obvious in command output

🤖 Prompt for AI agents

Create learning artifacts to help future agent runs discover and apply the CI
verification pattern.

Create `.serena/memories/skill-pr-007-ci-verification-required.md`:
- Follow existing skill memory format
- Document pattern: verify CI checks via `gh pr checks` before claiming PR
review completion and output evidence to session log
- Explain why this matters (mergeable != CI passing)
- Add evidence section showing Phase 8.1 verification requirement
- Set high atomicity rating (90%+) as mandatory requirement
- Include impact rating showing prevention of false completion claims

Update `.serena/memories/skills-pr-review-index.md`:
- Add entry for skill-pr-007
- Add keyword mapping: "CI checks verification mergeable status completion Phase
8.1"
- Group with completion verification skills (skill-pr-003)

Update `.claude/commands/pr-review.md` Step 6:
- Add CI check status per PR to summary generation
- Add table column or summary line showing CI status
- Use Phase 8.1 evidence format for visibility

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Restructure Phase 8 in the pr-comment-responder template to make CI verification
the first blocking gate.

In `templates/agents/pr-comment-responder.shared.md`:
- Renumber existing Phase 8.1 to Phase 8.2 (bump all sub-phases down)
- Create new Phase 8.1 "CI Status Verification" as first mandatory gate
- Add BLOCKING marker with instruction to not proceed if CI is failing
- Include `gh pr checks [number] --json name,state,conclusion` command
- Document that skipped checks count as passing
- Add required output format: "CI Status: [passing|failing] - [N] checks
successful, [M] checks skipped, [P] checks failed"
- Include instruction to list failing check names explicitly
- Add instruction to write CI status to task list with TodoWrite
- Update Phase 8.5 completion criteria to add CI verification evidence column
- Change "CI checks pass" criterion to reference Phase 8.1 output
- Add note that completion without Phase 8.1 evidence is a protocol violation
===============================================================================

Task: 2

Add validation script enforcement to verify CI check evidence in PR review
sessions.

In `scripts/Validate-SessionEnd.ps1`:
- Create `Test-PRCIVerification` function after existing test functions
- Detect PR review sessions by checking for pr-comment-responder agent reference
- Search session log for "CI Status:" evidence pattern
- Parse status line to verify passing/skipped checks (not failing)
- Return validation result with error if evidence missing or shows failures
- Integrate CI verification test into main validation sequence with other MUST
requirements
- Only run for pr-comment-responder sessions
- Make it a MUST requirement that blocks session end
- Add error message: "PR review session missing CI verification evidence in
Phase 8.1"

In `.agents/SESSION-END-PROMPT.md`:
- Add CI verification as mandatory gate for PR review sessions
- Include example of expected CI status evidence format
- Reference Phase 8.1 of pr-comment-responder
===============================================================================

Task: 3

Create learning artifacts to help future agent runs discover and apply the CI
verification pattern.

Create `.serena/memories/skill-pr-007-ci-verification-required.md`:
- Follow existing skill memory format
- Document pattern: verify CI checks via `gh pr checks` before claiming PR
review completion and output evidence to session log
- Explain why this matters (mergeable != CI passing)
- Add evidence section showing Phase 8.1 verification requirement
- Set high atomicity rating (90%+) as mandatory requirement
- Include impact rating showing prevention of false completion claims

Update `.serena/memories/skills-pr-review-index.md`:
- Add entry for skill-pr-007
- Add keyword mapping: "CI checks verification mergeable status completion Phase
8.1"
- Group with completion verification skills (skill-pr-003)

Update `.claude/commands/pr-review.md` Step 6:
- Add CI check status per PR to summary generation
- Add table column or summary line showing CI status
- Use Phase 8.1 evidence format for visibility

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!