bug: Verdict parsing fails when AI outputs VERDICT: [PASS] with brackets #575
Description
Summary
The verdict parsing regex in ai-review action fails to extract verdicts when the AI includes brackets around the verdict token (e.g., VERDICT: [PASS] instead of VERDICT: PASS).
Root Cause Analysis
The Problem
-
Prompt templates show options with brackets: Files like
spec-trace-requirements.md(line 54) andspec-check-completeness.md(line 60) show:VERDICT: [PASS|PARTIAL|FAIL] -
AI interprets brackets literally: Instead of outputting
VERDICT: PASS, the AI outputsVERDICT: [PASS]including the brackets. -
Regex can't match brackets: The verdict parsing in
action.ymlline 705:verdict=$(echo "$output" | sed -n 's/.*VERDICT:[[:space:]]*\([A-Z_]*\).*/\1/p' | tail -n 1)This regex
[A-Z_]*cannot match[so when parsingVERDICT: [PASS]:- After
VERDICT:there's a space (matched by[[:space:]]*) - The capture group tries to match from
[but[is not in[A-Z_] - Capture group is empty
- Falls through to
NEEDS_REVIEW(line 721)
- After
Evidence
From PR #543 CI run 20588542269:
- AI Output:
VERDICT: [PASS]with requirements traceability showing 100% coverage - Parsed verdict:
TRACE_VERDICT: NEEDS_REVIEW - Result: Job failed despite all requirements being satisfied
Proposed Fixes
Option A: Fix the regex (Quick fix)
Update line 705 to handle optional brackets:
verdict=$(echo "$output" | sed -n 's/.*VERDICT:[[:space:]]*\[\?\([A-Z_]*\)\]\?.*/\1/p' | tail -n 1)Option B: Fix the prompts (Root cause)
Update prompt templates to show expected output without the OR-style bracket notation:
# Before (confusing)
VERDICT: [PASS|PARTIAL|FAIL]
# After (explicit)
**VERDICT**: PASS, PARTIAL, or FAIL (use exact keyword, no brackets)
Example: VERDICT: PASSOption C: Both (Recommended)
- Fix regex for defense-in-depth
- Update prompts for clarity
Affected Files
.github/actions/ai-review/action.yml(parsing).github/prompts/spec-trace-requirements.md(prompt template).github/prompts/spec-check-completeness.md(prompt template)- Potentially other prompts using
[OPTION1|OPTION2]notation
Impact
- PRs with PASS verdicts incorrectly fail
- Requires manual investigation to determine actual status
- False negatives reduce trust in CI