Skip to content

Add inference-time threat evidence critic#26

Merged
arthurpanhku merged 2 commits into
mainfrom
codex/evidence-critic-threat-model
Jul 24, 2026
Merged

Add inference-time threat evidence critic#26
arthurpanhku merged 2 commits into
mainfrom
codex/evidence-critic-threat-model

Conversation

@arthurpanhku

@arthurpanhku arthurpanhku commented Jul 24, 2026

Copy link
Copy Markdown
Owner

What changed

  • add an inference-time Evidence Critic for Design/STRIDE threat findings, using one batched verifier call, stable current-document passage IDs, exact locators, and fail-safe abstention
  • normalize threat-model output and extend the report contract with support, contradiction, insufficient-evidence, confidence, and mandatory human-review fields
  • add a public threat-evidence panel plus an architecture walkthrough for the demo path
  • add grounding metrics, contract coverage, backend/frontend tests, and evaluation tests in CI
  • update React Router to the patched 7.18.1 release required by the production dependency audit

Why

DocSentinel could previously emit threat citations without independently checking whether the cited document text actually supported each threat claim. This focused vertical slice makes claim-level evidence visible and measurable without training or fine-tuning.

Impact

  • pure inference-time feature; no model training or fine-tuning required
  • one additional batched inference call for eligible Design assessments when enabled
  • all verifier verdicts remain advisory and require human review
  • verifier or parse failures degrade to insufficient_evidence

Validation

  • .venv/bin/ruff check .
  • .venv/bin/pytest -q — 117 passed
  • .venv/bin/pytest -q evals/tests — 5 passed
  • .venv/bin/python scripts/export_contracts.py --check
  • npm ci --prefix frontend
  • npm audit --omit=dev --prefix frontend — 0 vulnerabilities
  • npm run check --prefix frontend — 4 files / 6 tests passed
  • npm run build --prefix frontend
  • browser smoke test of client navigation, deep-link refresh, all three verdict states, and exact evidence locators

@socket-security

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updatednpm/​react-router-dom@​6.30.4 ⏵ 7.18.1100 +3100 +265 -995100

View full report

@arthurpanhku
arthurpanhku marked this pull request as ready for review July 24, 2026 16:02
@arthurpanhku
arthurpanhku merged commit 585c042 into main Jul 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant