Skip to content

Fix severity classification and agent scope from RI CCAP lessons#106

Merged
hua7450 merged 1 commit intoPolicyEngine:mainfrom
hua7450:ri-ccap-pipeline-lessons
Mar 5, 2026
Merged

Fix severity classification and agent scope from RI CCAP lessons#106
hua7450 merged 1 commit intoPolicyEngine:mainfrom
hua7450:ri-ccap-pipeline-lessons

Conversation

@hua7450
Copy link
Collaborator

@hua7450 hua7450 commented Mar 5, 2026

Summary

Fixes multiple pipeline issues discovered during the RI CCAP implementation session where correct findings were under-classified and agents exceeded their scope.

Root Causes Addressed

1. Severity classification gaps

The review pipeline correctly identified issues but classified them below CRITICAL, so the fix round skipped them:

  • Wrong section citations (e.g., 4.3.1(B)4.3.1(A)(4)) classified as medium
  • Hard-coded age values (1.5, 3, 6) classified as moderate
  • Untested complex formula variables classified as SHOULD ADDRESS
  • Non-functional tests (absolute_error_margin: 1 on booleans) classified as minor

2. Agent scope violations

  • Parameter-architect created variables and tests (not its job)
  • Orchestrator skipped rules-engineer and test-creator because files already existed
  • Result: lower-quality variables (bare input enums) and tests (wrong error margins, mismatched member counts)

3. Missing verification steps

  • Consolidator didn't verify citations against PDF text before writing impl-spec
  • Reference-checker proposed replacement citations without verifying them
  • No agent discovered existing reusable variables (e.g., childcare_hours_per_week)

Changes (10 files, 175 insertions)

Skills (shared knowledge — all agents inherit)

  • policyengine-parameter-patterns-skill — "Above X%" bracket boundary pattern (0.0001 shift)
  • policyengine-testing-patterns-skill — Type-specific error margins (none for booleans, 0.01 for currency); don't exhaustively test lookup tables
  • policyengine-variable-patterns-skill — Enum variables with numeric-range labels must have formulas using bracket parameters

Agents

  • reference-validator — Wrong section citations → CRITICAL; verify replacement citations against PDF text
  • implementation-validator — Hard-coded values labeled CRITICAL in report template; empty directory cleanup
  • parameter-architect — Explicit scope boundary: parameter YAML files ONLY
  • edge-case-generator — Bracket boundary consistency; representative testing not exhaustive
  • test-creator — Don't exhaustively test lookup tables

Commands

  • encode-policy-v2 — Consolidator verifies citations (Step 3) and discovers existing variables (Step 2); reporter includes Historical Notes; orchestrator always spawns rules-engineer + test-creator
  • review-program — Consolidator CRITICAL list expanded: incorrect section citations, untested formula variables, non-functional tests

🤖 Generated with Claude Code

…lessons

Key changes:
- Elevate wrong section citations to CRITICAL in reference-validator and consolidator
- Add citation verification against PDF text in consolidator (encode-policy-v2)
- Require reference-checker to verify replacement citations against source text
- Add empty directory cleanup to implementation-validator cross-reference check
- Add Historical Notes section to reporter PR description template
- Add "Above X%" bracket boundary pattern (0.0001 shift) to parameter-patterns skill
- Add bracket boundary consistency rule to edge-case-generator
- Add "don't exhaustively test lookup tables" to testing-patterns skill, test-creator, edge-case-generator
- Elevate hard-coded values (except 0, 1, 2) to CRITICAL in implementation-validator report
- Add enum-with-numeric-ranges must-have-formula pattern to variable-patterns skill
- Add existing variable discovery step to consolidator (encode-policy-v2)
- Add explicit scope boundary to parameter-architect (parameters ONLY)
- Add orchestrator rule to always spawn rules-engineer and test-creator
- Elevate untested formula variables to CRITICAL in consolidator
- Add non-functional tests (error margin >= 1 on booleans) to CRITICAL
- Replace blanket error margin guidance with type-specific rules

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@hua7450 hua7450 merged commit 7eab067 into PolicyEngine:main Mar 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant