feat(prompts): add validation review prompt for N+1 agent pass

[claude]

Summary

This PR implements an N+1 agent validation pass for the Claude Code PR Review workflow. When enabled, a second agent reviews and validates the first agent's findings before posting to GitHub.

What this does

• First Agent (Reviewer): Performs the initial PR review and outputs structured JSON
• Second Agent (Validator): Reviews the first agent's output, validates findings, and refines as needed

Benefits

Validation Check	Purpose
Verdict Accuracy	Catches overly harsh REQUEST_CHANGES for non-blocking issues, or APPROVE when bugs exist
False Positive Detection	Removes inline comments that flag correct code as incorrect
Missed Issue Detection	Adds comments for bugs/security issues the first agent missed
Line Number Validation	Verifies inline comments reference lines within the diff
Suggestion Accuracy	Ensures code suggestions are syntactically correct

Usage

uses: Uniswap/ai-toolkit/.github/workflows/_claude-code-review.yml@main
with:
  pr_number: ${{ github.event.pull_request.number }}
  base_ref: ${{ github.base_ref }}
  model: 'claude-opus-4-5'
  enable_validation_pass: true
  validation_model: 'claude-sonnet-4-5'  # Optional: different model for validation
secrets:
  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Cost/Time Tradeoffs

• Approximately doubles review time and API costs
• Recommended for critical repositories or when review accuracy is paramount
• Consider using a faster/cheaper model for validation (e.g., Sonnet) while using Opus for the initial review

Changes

Committed (prompt files)

• Added: .github/prompts/validation-review.md - Validation agent prompt
• Modified: .github/prompts/CLAUDE.md - Documentation for new prompt

Requires Manual Application (workflow files)

Due to GitHub App permission restrictions, workflow file changes must be applied manually.

The following changes need to be applied to complete this feature:

Click to expand workflow changes (.github/workflows/_claude-code-review.yml)

New inputs to add after toolkit_ref (around line 106):

      enable_validation_pass:
        description: "Enable N+1 agent validation: a second agent reviews and validates the first agent's findings before posting. This improves review quality by catching errors, false positives, and missed issues."
        required: false
        type: boolean
        default: false

      validation_model:
        description: "Claude model to use for the validation pass. Defaults to the same model as the review."
        required: false
        type: string
        default: ""

New steps to add after "Upload Claude Execution Output" step (around line 815):

      # ==========================================================================
      # N+1 VALIDATION PASS (Optional)
      # When enabled, a second agent reviews the first agent's output before posting.
      # This helps catch false positives, incorrect verdicts, and missed issues.
      # ==========================================================================

      # Build validation prompt with first agent's review
      - name: Build Validation Prompt
        if: steps.cache-check.outputs.cache-hit != 'true' && success() && inputs.enable_validation_pass == true
        id: build-validation-prompt
        env:
          TOOLKIT_REF: ${{ inputs.toolkit_ref }}
          REPO: ${{ github.repository }}
          PR_NUMBER: ${{ inputs.pr_number }}
          BASE_REF: ${{ steps.pr-info.outputs.base_ref }}
          MERGE_BASE: ${{ steps.patch-id.outputs.merge_base }}
        run: |
          set -euo pipefail

          echo "ℹ️  Building validation prompt for N+1 agent pass..."

          # Fetch validation prompt from ai-toolkit repository
          if [ "${{ github.repository }}" = "Uniswap/ai-toolkit" ]; then
            VALIDATION_PROMPT=$(cat .github/prompts/validation-review.md)
            echo "ℹ️  Using local validation prompt"
          else
            echo "ℹ️  Fetching validation prompt from Uniswap/ai-toolkit@${TOOLKIT_REF}"
            if ! curl -fSs \
                      -H "Accept: application/vnd.github.v4.raw" \
                      -L "https://api.github.com/repos/Uniswap/ai-toolkit/contents/.github/prompts/validation-review.md?ref=${TOOLKIT_REF}" \
                      -o /tmp/validation-prompt.md 2>&1 | tee /tmp/curl-error.log; then
              echo "❌ Failed to fetch validation prompt"
              cat /tmp/curl-error.log | head -5
              exit 1
            fi
            VALIDATION_PROMPT=$(cat /tmp/validation-prompt.md)
          fi

          # Parse repository owner and name
          REPO_OWNER=$(echo "$REPO" | cut -d'/' -f1)
          REPO_NAME=$(echo "$REPO" | cut -d'/' -f2)

          # Build the validation prompt with context
          cat > /tmp/validation-final-prompt.txt <<VALIDATION_PROMPT_EOF
          # Validation Context

          **Repository Owner:** $REPO_OWNER
          **Repository Name:** $REPO_NAME
          **PR Number:** $PR_NUMBER
          **Base Branch:** $BASE_REF



          # First Agent's Review Output

          The first agent produced the following review. Your task is to validate and potentially refine it.

          \`\`\`json
          $(cat /tmp/claude-review-response.txt)
          \`\`\`



          # PR Diff (for reference)

          **To get the correct diff, use this command:**
          \`\`\`bash
          git diff ${MERGE_BASE}..HEAD
          \`\`\`

          **Files changed in this PR:**
          \`\`\`
          $(cat /tmp/changed-files.txt)
          \`\`\`

          **Full PR Diff:**
          \`\`\`diff
          $(cat /tmp/pr-diff.txt)
          \`\`\`



          $VALIDATION_PROMPT
          VALIDATION_PROMPT_EOF

          # Log prompt size
          PROMPT_LENGTH=$(wc -c < /tmp/validation-final-prompt.txt)
          echo "✅ Built validation prompt: $PROMPT_LENGTH characters"

          # Export for next step
          {
            echo "validation_prompt<<VALIDATION_PROMPT_DELIMITER_EOF"
            cat /tmp/validation-final-prompt.txt
            echo "VALIDATION_PROMPT_DELIMITER_EOF"
          } >> $GITHUB_OUTPUT

      # Build Claude arguments for validation pass
      - name: Build Validation Arguments
        if: steps.cache-check.outputs.cache-hit != 'true' && success() && inputs.enable_validation_pass == true
        id: build-validation-args
        env:
          INPUT_MODEL: ${{ inputs.model }}
          VALIDATION_MODEL: ${{ inputs.validation_model }}
        run: |
          # Use validation_model if specified, otherwise use the same model as review
          if [ -n "$VALIDATION_MODEL" ]; then
            MODEL="$VALIDATION_MODEL"
            echo "ℹ️  Using validation model: $MODEL"
          else
            MODEL="$INPUT_MODEL"
            echo "ℹ️  Using review model for validation: $MODEL"
          fi

          # Same read-only tools as the review pass
          TOOLS=(
            "Read"
            "Grep"
            "Glob"
            "Bash(git log:*)"
            "Bash(git diff:*)"
            "Bash(git show:*)"
            "Bash(git blame:*)"
            "Bash(git rev-parse:*)"
            "mcp__github__get_pull_request"
            "mcp__github__get_pull_request_files"
            "mcp__github__get_pull_request_comments"
            "mcp__github__get_pull_request_reviews"
          )
          ALLOWED_TOOLS=$(IFS=,; echo "${TOOLS[*]}")

          # Same JSON schema as review pass
          JSON_SCHEMA='{
            "type": "object",
            "properties": {
              "pr_review_body": {
                "type": "string",
                "description": "Markdown-formatted review summary with all analysis"
              },
              "pr_review_outcome": {
                "type": "string",
                "enum": ["COMMENT", "APPROVE", "REQUEST_CHANGES"],
                "description": "Review decision"
              },
              "inline_comments_new": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "path": { "type": "string", "description": "File path" },
                    "line": { "type": "integer", "description": "Line number" },
                    "body": { "type": "string", "description": "Comment text" },
                    "suggestion": { "type": "string", "description": "Optional code suggestion" }
                  },
                  "required": ["path", "line", "body"]
                },
                "description": "New inline comments on specific lines"
              },
              "inline_comments_responses": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "comment_id": { "type": "integer", "description": "ID of existing comment" },
                    "body": { "type": "string", "description": "Response text" },
                    "should_resolve": { "type": "boolean", "description": "Whether to resolve the thread" }
                  },
                  "required": ["comment_id", "body"]
                },
                "description": "Responses to existing review comments"
              },
              "files_reviewed": {
                "type": "array",
                "items": { "type": "string" },
                "description": "List of files reviewed"
              },
              "confidence": {
                "type": "number",
                "minimum": 0,
                "maximum": 1,
                "description": "Confidence level (0.0-1.0)"
              }
            },
            "required": ["pr_review_body", "pr_review_outcome", "inline_comments_new"]
          }'

          COMPACT_SCHEMA=$(echo "$JSON_SCHEMA" | jq -c .)
          VALIDATION_ARGS="--model $MODEL
            --json-schema '$COMPACT_SCHEMA'
            --allowedTools $ALLOWED_TOOLS"

          echo "validation_args<<VALIDATION_ARGS_EOF" >> $GITHUB_OUTPUT
          echo "$VALIDATION_ARGS" >> $GITHUB_OUTPUT
          echo "VALIDATION_ARGS_EOF" >> $GITHUB_OUTPUT
          echo "✅ Configured validation pass arguments"

      # Run validation agent (second pass)
      - name: Run Validation Agent
        if: steps.cache-check.outputs.cache-hit != 'true' && success() && inputs.enable_validation_pass == true
        id: validation
        uses: anthropics/claude-code-action@a7e4c51380c42dd89b127f5e5f9be7b54020bc6b # v1.0.21
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          prompt: ${{ steps.build-validation-prompt.outputs.validation_prompt }}
          track_progress: false
          show_full_output: false
          allowed_bots: dependabot
          claude_args: ${{ steps.build-validation-args.outputs.validation_args }}
          additional_permissions: |
            actions: read

      # Extract validation agent's output
      - name: Extract Validation Response
        if: steps.cache-check.outputs.cache-hit != 'true' && success() && inputs.enable_validation_pass == true
        id: extract-validation-response
        env:
          STRUCTURED_OUTPUT: ${{ steps.validation.outputs.structured_output }}
        run: |
          set -euo pipefail

          echo "ℹ️  Extracting validation agent's output..."

          if [ -z "$STRUCTURED_OUTPUT" ]; then
            echo "❌ No structured_output from validation agent"
            echo "   Falling back to first agent's review"
            echo "use_validation=false" >> $GITHUB_OUTPUT
            exit 0
          fi

          # Save validation output (overwrites first agent's response)
          echo "$STRUCTURED_OUTPUT" > /tmp/claude-review-response.txt
          echo "use_validation=true" >> $GITHUB_OUTPUT
          echo "✅ Validation pass complete - using validated review"

          # Show comparison preview
          echo "📋 Validated response preview:"
          echo "$STRUCTURED_OUTPUT" | jq -r '.pr_review_outcome // "unknown"' | xargs -I{} echo "  Outcome: {}"
          echo "$STRUCTURED_OUTPUT" | jq -r '.inline_comments_new | length' | xargs -I{} echo "  Inline comments: {}"
          echo "$STRUCTURED_OUTPUT" | jq -r '.confidence // "not set"' | xargs -I{} echo "  Confidence: {}"

      # Upload validation output as artifact for debugging
      - name: Upload Validation Output
        if: steps.cache-check.outputs.cache-hit != 'true' && inputs.enable_validation_pass == true && always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: claude-validation-output-pr${{ inputs.pr_number }}
          path: /home/runner/work/_temp/claude-execution-output.json
          retention-days: 7
          if-no-files-found: warn

Click to expand consumer workflow changes (.github/workflows/claude-code-review.yml)

Add documentation comments after the model: line in each job:

      # N+1 Validation Pass: Enable a second agent to validate the first agent's review
      # This helps catch false positives, incorrect verdicts, and missed issues
      # Note: This approximately doubles review time and API costs
      # Uncomment to enable:
      # enable_validation_pass: true
      # validation_model: 'claude-sonnet-4-5'  # Optional: use a different model for validation

Click to expand documentation changes (.github/workflows/CLAUDE.md)

See the committed changes to .github/prompts/CLAUDE.md for documentation pattern. Similar documentation should be added to .github/workflows/CLAUDE.md once the workflow changes are applied.

Testing

Once the workflow changes are manually applied:

1. Enable the validation pass in a test PR:

   enable_validation_pass: true

2. Verify:

   • First agent produces initial review JSON
   • Validation agent runs and reviews the output
   • Final review reflects validated/refined findings
   • Both artifacts are uploaded for debugging

Linear Issue

Closes: https://linear.app/uniswap/issue/DEV-167/add-n1-agent-configuration-for-claude-code-pr-review-github-actions

Autonomous implementation using claude-opus-4-5-20251101

---

✨ Claude-Generated Content

Summary

Adds validation review prompt for the N+1 agent validation pass in Claude Code PR reviews. When enabled, a second agent reviews and validates the first agent's findings before posting to GitHub.

Changes

• Added .github/prompts/validation-review.md - Comprehensive validation agent guidelines covering:
  • Verdict accuracy validation (catching overly harsh REQUEST_CHANGES or missed bugs)
  • Inline comment validity checks (line numbers, false positives, suggestion syntax)
  • Completeness verification (finding bugs/security issues the first agent missed)
  • Tone and actionability standards
  • Decision rules for when to upgrade/downgrade verdicts or add/remove comments
• Updated .github/prompts/CLAUDE.md - Added documentation for the new validation prompt

Notes

This PR contains only the prompt files. The corresponding workflow changes (_claude-code-review.yml) that enable the enable_validation_pass and validation_model inputs must be applied manually due to GitHub App permission restrictions. See the PR description for the full workflow integration details.

Linear Issue

Closes: DEV-167

[linear]

DEV-167 Add N+1 Agent configuration for Claude Code PR Review GitHub Actions workflow

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
ai-toolkit-slack-oauth-backend	Ready	Preview, Comment	Dec 12, 2025 11:39pm

[github-actions]

👋 Hi!

I'm Claude, an AI assistant here to help with code reviews and answer questions about your PR. You can tag me anytime with @claude followed by your request.

💡 Tip: Ask me to explain code, suggest improvements, review specific changes, or help with AI Toolkit development patterns.

This repository contains the AI Toolkit - standardized, one-shot setup for Claude Code AI workflows. When working on changes, make sure to follow the Nx monorepo patterns and update relevant CLAUDE.md files.

Learn how to use Claude in ai-toolkit PRs

[github-actions]

🤖 Claude Code Review

🔄 Review in progress... Claude is analyzing this pull request. If a review can be seen below, it will be replaced by the results of this one.

_{⏱️ Reviews typically complete within 5-15 minutes depending on PR size.}

Waiting for review to complete...

---

_{💡 Want a fresh review? Add a comment containing @request-claude-review to trigger a new review at any time.}

[github-actions]

🤖 Claude Code Review

Status: ⚠️ Partial (script error)
Job: View workflow run

---

The review was completed but there was an error posting inline comments.

Raw Review Output

No review output available

---

Please check the workflow logs for details.