Attestation Evidence Audit — Main Tracker

[AlexKantor87]

Attestation Evidence Audit — Main Tracker

Systematic review of all 31 attestation types across the agentic SDLC pipeline. Each ticket gets multi-persona analysis (Pipeline Engineer, Security Architect, Compliance Officer, Reliability Engineer) followed by critical audit review (Internal Auditor, Red Team, AI Governance Specialist).

Process

1. ✅ Template agreed
2. ✅ All 31 sub-tickets created with template headings
3. ✅ Pilot: Fill + audit artifact-integrity-control end-to-end
4. ✅ Review pilot quality, refine approach
5. ✅ Scale to remaining 30 tickets
6. ✅ Synthesis: summary table + remediation backlog (see comments below)

---

Summary Statistics

Metric	Count
Total attestation types audited	31
PASS (clean)	4
PASS WITH OBSERVATIONS	5
CONDITIONAL / QUALIFIED PASS	3
ADEQUATE (various)	7
SOUND WITH KNOWN LIMITATIONS	1
NEEDS IMPROVEMENT	9
INADEQUATE	0
Total gaps identified	~140
P1 recommendations	6
Code bugs found	7

Strongest group: AI Review Steps (Group D) — 4 clean PASSes, well-structured schemas and jq rules.
Weakest group: Build Flow Controls (Group F) — 6/7 NEEDS IMPROVEMENT, all use type: generic bypassing server-side validation.

---

Group A: Review Flow — CI Gate

• [Audit] #1 pre-review-lint #102 — pre-review-lint — ✅ ADEQUATE
• [Audit] #2 pre-review-unit-tests #103 — pre-review-unit-tests — ✅ ADEQUATE
• [Audit] #3 pre-review-integration-tests #104 — pre-review-integration-tests — ✅ ADEQUATE
• [Audit] #4 pre-review-docker-build #105 — pre-review-docker-build — ⚠️ ADEQUATE WITH CAVEATS

Group B: Review Flow — Orchestration

• [Audit] #5 change-classified #106 — change-classified — ✅ PASS WITH OBSERVATIONS
• [Audit] #6 prior-context-built #107 — prior-context-built — ✅ PASS WITH OBSERVATIONS
• [Audit] #7 findings-deduplicated #108 — findings-deduplicated — ✅ PASS WITH OBSERVATIONS
• [Audit] #8 cross-model-severity-applied #109 — cross-model-severity-applied — 🔶 NEEDS IMPROVEMENT
• [Audit] #9 moderator-debate #110 — moderator-debate — 🔶 NEEDS IMPROVEMENT
• [Audit] #10 moderator-resolution #111 — moderator-resolution — 🔶 NEEDS IMPROVEMENT

Group C: Review Flow — Resolver

• [Audit] #11 resolver-threads-fetched #112 — resolver-threads-fetched — ✅ ADEQUATE
• [Audit] #12 resolver-threads-triaged #113 — resolver-threads-triaged — ✅ ADEQUATE WITH OBSERVATIONS
• [Audit] #13 resolver-fixes-committed #114 — resolver-fixes-committed — 🔶 NEEDS IMPROVEMENT
• [Audit] #14 resolver-threads-resolved #115 — resolver-threads-resolved — 🔶 NEEDS IMPROVEMENT

Group D: Review Flow — AI Review Steps

• [Audit] #15 standards-loaded #116 — standards-loaded — ✅ PASS
• [Audit] #16 prior-context-applied #117 — prior-context-applied — ✅ PASS
• [Audit] #17 diff-received #118 — diff-received — ✅ PASS
• [Audit] #18 review-executed #119 — review-executed — ⚠️ CONDITIONAL PASS
• [Audit] #19 praise-filtered #120 — praise-filtered — ✅ PASS WITH OBSERVATIONS
• [Audit] #20 findings-structured #121 — findings-structured — ✅ PASS
• [Audit] #21 severity-justified #122 — severity-justified — ✅ PASS WITH OBSERVATIONS
• [Audit] #22 review-summary #123 — review-summary — ✅ PASS

Group E: Review Flow — Governance

• [Audit] #23 ticket-integrity #124 — ticket-integrity 🔴 — ✅ ADEQUATE
• [Audit] #24 loop-cost #125 — loop-cost — ⚠️ QUALIFIED PASS
• [Audit] #25 reviewed-code artifact identity #126 — reviewed-code artifact identity 🔴 — ✅ SOUND WITH KNOWN LIMITATIONS

Group F: Build Flow

• [Audit] #26 coding-agent-completed #127 — coding-agent-completed 🔴 — 🔶 NEEDS IMPROVEMENT
• [Audit] #27 ci-docker-build (build flow) #128 — ci-docker-build (build flow) 🔴 — 🔶 NEEDS IMPROVEMENT
• [Audit] #29 ticket-integrity-control #129 — ticket-integrity-control 🔴 — ⚠️ CONDITIONAL PASS
• [Audit] Build Flow Controls: lint-control, unit-test-control, integration-test-control, cost-control #130 — lint/unit-test/integration-test/cost controls 🔴 — 🔶 NEEDS IMPROVEMENT
• [Audit] #28 code-review-control #131 — code-review-control 🔴 — 🔶 NEEDS IMPROVEMENT
• [Audit] #31 artifact-integrity-control #132 — artifact-integrity-control 🔴 — 🔶 NEEDS IMPROVEMENT

🔴 = priority:high (compliance-critical)

---

Known Issues (Pre-Audit)

1. ✅ Confirmed: Build flow controls use generic type — custom types with jq rules exist but aren't used
2. ✅ Confirmed: 4 build controls have no custom type at all (lint, unit-test, integration-test, cost)
3. ✅ Confirmed: pre-review-docker-build is generic — no schema/jq validation on source_fingerprint
4. ✅ Confirmed: resolver-threads-triaged has self-attestation risk (agent sets its own compliance field)
5. ✅ Confirmed: Ticket integrity lock relies on forgeable comment string matching

New Issues (Discovered During Audit)

6. severities_updated hardcoded to 0 in cross-model-severity ([Audit] #8 cross-model-severity-applied #109)
7. OpenAI excluded from moderator debate analysis ([Audit] #9 moderator-debate #110)
8. Schema/jq enum mismatch for APPROVE_WITH_COMMENTS ([Audit] #10 moderator-resolution #111)
9. tokens_used hardcoded to 0, truncated field missing in review-executed ([Audit] #18 review-executed #119)
10. findings_removed hardcoded to 0 in praise-filtered ([Audit] #19 praise-filtered #120)
11. No-op jq rule .high_findings_with_standards_ref >= 0 always true in severity-justified ([Audit] #21 severity-justified #122)
12. commit_succeeded attested BEFORE git commit happens — records intent not outcome ([Audit] #13 resolver-fixes-committed #114)
13. summary_comment_posted hardcoded to true — control provides no real assurance ([Audit] #14 resolver-threads-resolved #115)
14. Shared KOSLI_API_TOKEN allows cross-flow attestation forgery ([Audit] #31 artifact-integrity-control #132)
15. .kosli_ignore manipulation as supply chain attack vector ([Audit] #25 reviewed-code artifact identity #126, [Audit] #31 artifact-integrity-control #132)
16. Shell injection via unquoted trail names in CI workflow ([Audit] #31 artifact-integrity-control #132)
17. No human-in-the-loop on happy path — AI authors, reviews, fixes, ships ([Audit] #31 artifact-integrity-control #132)

[AlexKantor87]

Attestation Evidence Audit — Synthesis Report

Executive Summary

31 attestation types were audited across two Kosli flows (Code Review: 96 slots/trail, Build: 9 slots/trail). Each was analysed by 4 fill-out personas (Pipeline Engineer, Security Architect, Compliance Officer, Reliability Engineer) and critically reviewed by 3 audit personas (SOC 2 Internal Auditor, Adversarial Red Team, AI Governance Specialist).

Overall assessment: The pipeline provides a solid foundation of audit evidence but has structural weaknesses in the build flow controls and several self-attestation gaps that need to be addressed before regulatory exposure.

The review flow's custom attestation types (with JSON schemas + jq evaluator rules) provide meaningful server-side validation. The build flow's reliance on type: generic bypasses all of this — it's the single largest systemic gap.

---

Verdict Summary Table

#	Ticket	Slot Name	Group	Verdict	P1 Gaps
1	#102	pre-review-lint	CI Gate	✅ ADEQUATE	0
2	#103	pre-review-unit-tests	CI Gate	✅ ADEQUATE	0
3	#104	pre-review-integration-tests	CI Gate	✅ ADEQUATE	0
4	#105	pre-review-docker-build	CI Gate	⚠️ ADEQUATE WITH CAVEATS	2
5	#106	change-classified	Orchestration	✅ PASS WITH OBS	0
6	#107	prior-context-built	Orchestration	✅ PASS WITH OBS	0
7	#108	findings-deduplicated	Orchestration	✅ PASS WITH OBS	0
8	#109	cross-model-severity-applied	Orchestration	🔶 NEEDS IMPROVEMENT	0
9	#110	moderator-debate	Orchestration	🔶 NEEDS IMPROVEMENT	0
10	#111	moderator-resolution	Orchestration	🔶 NEEDS IMPROVEMENT	0
11	#112	resolver-threads-fetched	Resolver	✅ ADEQUATE	0
12	#113	resolver-threads-triaged	Resolver	✅ ADEQUATE WITH OBS	0
13	#114	resolver-fixes-committed	Resolver	🔶 NEEDS IMPROVEMENT	2
14	#115	resolver-threads-resolved	Resolver	🔶 NEEDS IMPROVEMENT	2
15	#116	standards-loaded	AI Review	✅ PASS	0
16	#117	prior-context-applied	AI Review	✅ PASS	0
17	#118	diff-received	AI Review	✅ PASS	0
18	#119	review-executed	AI Review	⚠️ CONDITIONAL PASS	0
19	#120	praise-filtered	AI Review	✅ PASS WITH OBS	0
20	#121	findings-structured	AI Review	✅ PASS	0
21	#122	severity-justified	AI Review	✅ PASS WITH OBS	0
22	#123	review-summary	AI Review	✅ PASS	0
23	#124	ticket-integrity	Governance	✅ ADEQUATE	0
24	#125	loop-cost	Governance	⚠️ QUALIFIED PASS	0
25	#126	reviewed-code artifact	Governance	✅ SOUND	0
26	#127	coding-agent-completed	Build	🔶 NEEDS IMPROVEMENT	0
27	#128	ci-docker-build (build)	Build	🔶 NEEDS IMPROVEMENT	0
28	#129	ticket-integrity-control	Build	⚠️ CONDITIONAL PASS	0
29	#130	lint/unit/integ/cost controls	Build	🔶 NEEDS IMPROVEMENT	0
30	#131	code-review-control	Build	🔶 NEEDS IMPROVEMENT	0
31	#132	artifact-integrity-control	Build	🔶 NEEDS IMPROVEMENT	0

---

Top Findings by Theme

Theme 1: Build Flow Uses generic Type — No Server-Side Validation

Affects: #127, #128, #129, #130, #131, #132 (all build flow controls)
Severity: HIGH
Description: All build flow artifact attestations use type: generic in build-template.yml, yet custom types with jq evaluator rules and JSON schemas exist in setup.sh for several of these controls. The generic type has zero server-side validation — any JSON payload is accepted. This means:

• An attacker who compromises the CI runner can forge compliant attestations with arbitrary data
• Evidence structure is not enforced — missing fields, wrong types, fabricated values all pass
• The carefully designed jq rules and schemas in setup.sh are completely bypassed
  Recommendation: Switch build flow attestation types from generic to their corresponding custom types. For the 4 controls without custom types (lint, unit-test, integration-test, cost), create new custom types with appropriate jq rules.

Theme 2: Self-Attestation Pattern Throughout

Affects: #113, #114, #115, #119, #120, #122, #127
Severity: MEDIUM-HIGH
Description: Multiple attestations are produced by the same process being audited. The resolver attests its own triage quality (#113), its own commit success (#114), and its own thread resolution (#115). The review agents attest their own execution metrics (#119). The coding agent attests its own completion (#127). In each case, a compromised or malfunctioning agent could report compliant evidence while actual behavior diverges.
Recommendation: Where feasible, move attestation production to an independent observer (e.g., the runner attests resolver results after independent verification). Where self-attestation is unavoidable, add cross-attestation consistency checks.

Theme 3: Hardcoded / No-Op Evidence Fields

Affects: #109, #115, #119, #120, #122
Severity: MEDIUM
Description: Several attestations contain fields that are always hardcoded to a fixed value, making the corresponding jq rules trivially satisfiable:

• severities_updated hardcoded to 0 ([Audit] #8 cross-model-severity-applied #109)
• summary_comment_posted hardcoded to true ([Audit] #14 resolver-threads-resolved #115)
• tokens_used hardcoded to 0 ([Audit] #18 review-executed #119)
• findings_removed hardcoded to 0 ([Audit] #19 praise-filtered #120)
• .high_findings_with_standards_ref >= 0 always passes ([Audit] #21 severity-justified #122)

These fields appear in the evidence but provide zero assurance. Auditors reviewing the trail would see them and assume they represent real measurements.
Recommendation: Either implement the actual measurement (preferred) or remove the field from the schema/jq rules until it can be implemented honestly. Hardcoded "always compliant" fields are worse than absent fields.

Theme 4: Timing / TOCTOU Issues in Resolver Chain

Affects: #114, #115
Severity: HIGH
Description: resolver-fixes-committed (#114) attests commit_succeeded: true before the git commit actually happens (resolver attests at line 262, runner commits at line 528). If the commit fails, the attestation is wrong and cannot be retracted. The commit_sha field contains the pre-fix SHA, not the actual commit SHA. This is the most critical evidence integrity issue found in the audit.
Recommendation: Move the resolver-fixes-committed attestation to the runner, attesting after _git_commit_fixes() returns with the actual commit SHA and success/failure result.

Theme 5: Cross-Flow Security Boundary Weakness

Affects: #132, #126
Severity: MEDIUM-HIGH
Description: The review flow and build flow share the same KOSLI_API_TOKEN, meaning a compromised review agent could attest to build flow slots (or vice versa). The review_flow_name parameter in verify_artifact_integrity.sh is not validated — an attacker could point it at any flow. The .kosli_ignore file could be manipulated to exclude files from the source fingerprint.
Recommendation: Use separate, scoped API tokens for review and build flows. Validate flow names against a whitelist. Consider making .kosli_ignore part of the fingerprinted content or using a policy-controlled exclusion list.

Theme 6: Orchestration Controls Missing Key Data

Affects: #109, #110, #111
Severity: MEDIUM
Description: The orchestration controls (cross-model severity, moderator debate, moderator resolution) have schema/jq mismatches and incomplete evidence:

• Cross-model severity doesn't track actual severity changes ([Audit] #8 cross-model-severity-applied #109)
• Moderator debate excludes OpenAI agent findings from analysis ([Audit] #9 moderator-debate #110)
• Moderator resolution has enum mismatch for APPROVE_WITH_COMMENTS ([Audit] #10 moderator-resolution #111)
  Recommendation: Fix the code bugs and ensure schemas/jq rules align with actual evidence produced.

---

Code Bugs Found During Audit

#	Location	Bug	Ticket
1	branch_orchestrator.py	severities_updated hardcoded to 0	#109
2	branch_orchestrator.py	OpenAI excluded from moderator debate analysis	#110
3	Schema + jq	APPROVE_WITH_COMMENTS enum mismatch	#111
4	base_orchestrator.py	tokens_used hardcoded to 0	#119
5	base_orchestrator.py	truncated field missing from attestation	#119
6	base_orchestrator.py	findings_removed hardcoded to 0	#120
7	setup.sh jq rule	.high_findings_with_standards_ref >= 0 is a no-op	#122

---

Verdict Distribution

PASS (clean)              ████░░░░░░  4  (13%)
PASS WITH OBSERVATIONS    █████░░░░░  5  (16%)
CONDITIONAL/QUALIFIED     ███░░░░░░░  3  (10%)
ADEQUATE (various)        ███████░░░  7  (23%)
SOUND WITH LIMITATIONS    █░░░░░░░░░  1  (3%)
NEEDS IMPROVEMENT         █████████░  9  (29%)
INADEQUATE                ░░░░░░░░░░  0  (0%)

62% of attestations are ADEQUATE or better. No attestation was rated INADEQUATE — the pipeline provides baseline evidence for all 105 slots. The 9 NEEDS IMPROVEMENT ratings are concentrated in the build flow (6) and specific weak points in orchestration/resolver (3).

---

Strengths

1. Source directory fingerprinting model is sound — deterministic, reproducible, enables cross-flow integrity verification
2. AI review step attestations are well-designed — 8-step pipeline with custom types, schemas, and jq rules for each step × 10 agents = 80 independently validated attestation slots
3. Conservation-of-count rules are strong — resolver-threads-triaged jq rule 1 is mathematically exact
4. Fail-safe on inference fallback — resolver blocks inferred_fallback decisions, forcing re-execution
5. Cost tracking with budget enforcement — within_budget check prevents runaway agent costs
6. Ticket integrity with lock/snapshot/verify — three-phase approach is architecturally sound even if implementation has forgery risks

[AlexKantor87]

Prioritised Remediation Backlog

Recommendations consolidated from all 31 audit tickets, ordered by impact and effort.

---

🔴 Priority 1 — Must Fix (Compliance-Critical)

R1. Switch build flow attestation types from generic to custom types

Effort: Medium (template + setup.sh changes, re-run pipeline)
Tickets: #127, #128, #129, #130, #131, #132
What: Update build-template.yml to use custom:coding-agent-result, custom:ci-docker-build, custom:code-review-control, custom:ticket-integrity-control for slots that already have custom types defined in setup.sh. Create new custom types for lint-control, unit-test-control, integration-test-control, cost-control with appropriate jq rules.
Why: Without this, all build flow evidence has zero server-side validation. Any JSON payload is accepted as compliant. This is the single largest systemic gap.

R2. Move resolver-fixes-committed attestation to after git commit

Effort: Small (code refactor in runner.py + branch_resolver.py)
Tickets: #114
What: The resolver currently attests commit_succeeded: true before the commit happens. Move attestation to the runner, after _git_commit_fixes() returns. Capture the actual post-commit SHA and success/failure result.
Why: The attestation currently records intent, not outcome. If the git commit fails, Kosli has a false-positive compliant attestation that cannot be retracted.

R3. Fix summary_comment_posted hardcoding in resolver-threads-resolved

Effort: Small (implement actual summary posting or remove the field)
Tickets: #115
What: Either implement GitHub summary comment posting and derive the boolean from the API response, or rename the attestation to resolver-triage-summary and remove the hardcoded boolean.
Why: The current jq rule summary_comment_posted == true provides zero assurance — it's always true regardless of what actually happened.

---

🟠 Priority 2 — Should Fix (Evidence Quality)

R4. Fix 7 code bugs found during audit

Effort: Small-Medium (7 individual fixes)
Tickets: #109, #110, #111, #119, #120, #122
What:

• Fix severities_updated to track actual severity changes ([Audit] #8 cross-model-severity-applied #109)
• Include OpenAI agents in moderator debate analysis ([Audit] #9 moderator-debate #110)
• Align APPROVE_WITH_COMMENTS across schema, jq, and code ([Audit] #10 moderator-resolution #111)
• Implement actual tokens_used tracking from API responses ([Audit] #18 review-executed #119)
• Add truncated field to review-executed attestation ([Audit] #18 review-executed #119)
• Implement actual findings_removed tracking in praise filter ([Audit] #19 praise-filtered #120)
• Replace no-op jq rule in severity-justified with meaningful check ([Audit] #21 severity-justified #122)

R5. Implement cross-flow token scoping

Effort: Medium (infrastructure + workflow changes)
Tickets: #132, #126
What: Use separate, scoped Kosli API tokens for review and build flows so a compromised review agent cannot attest to build flow slots.
Why: Shared tokens mean the security boundary between flows exists only in convention, not in enforcement.

R6. Detect iteration limit exhaustion in resolver

Effort: Small (parse Claude Code CLI output)
Tickets: #113
What: Parse Claude Code CLI output for turn-limit signals and set iteration_limit_reached accordingly. Currently hardcoded to False, making jq rule 3 a no-op.
Why: The jq rule is well-designed but never triggers because the field never changes.

R7. Add count reconciliation across resolver chain

Effort: Small (add fields to attestation payloads)
Tickets: #112, #113, #114, #115
What: Add expected_threads field to triage attestation so threads_triaged == unresolved_threads_found from the upstream fetch can be verified. Add total_review_findings to the fetch attestation.
Why: Currently each resolver attestation is internally consistent but there's no cross-attestation invariant enforcement.

---

🟡 Priority 3 — Should Consider (Hardening)

R8. Validate review_flow_name parameter in artifact integrity check

Effort: Small
Tickets: #132
What: Validate that the flow name matches the expected pattern agentic-sdlc-demo-GH{issue}-CodeReview before querying Kosli.
Why: Prevents flow spoofing where an attacker points the integrity check at a different (clean) flow.

R9. Add shell injection protection in CI workflow

Effort: Small (quote variables in ci.yml)
Tickets: #132
What: Quote all variable expansions in shell commands within ci.yml, particularly trail names and flow names that contain user-controllable input.
Why: Unquoted variables with user-controllable issue numbers could enable command injection.

R10. Add findings_filter_applied flag to resolver-threads-fetched

Effort: Small
Tickets: #112
What: Add a boolean field indicating when the resolver received a filtered subset of findings (e.g., only HIGH/MEDIUM in final round).
Why: Auditors comparing unresolved_threads_found across attestations may be confused when the count differs from total review findings.

R11. Strengthen ticket integrity against forgery

Effort: Medium
Tickets: #124
What: Replace comment string matching ("locked for the duration" / "Issue unlocked") with GitHub's native lock API. Use cryptographic hashes or signed payloads for body/comment snapshots.
Why: Current lock mechanism relies on string markers that anyone with write access can forge.

R12. Add .kosli_ignore content to fingerprint evidence

Effort: Small
Tickets: #126, #132
What: Include the contents of .kosli_ignore in the attestation payload so auditors can verify what was excluded from the fingerprint.
Why: A manipulated .kosli_ignore could exclude malicious files from the fingerprint without detection.

---

🔵 Priority 4 — Future Consideration (Design)

R13. Introduce independent attestation observer for resolver chain

Effort: Large (architecture change)
Tickets: #113, #114, #115
What: Add a lightweight independent process that verifies resolver outcomes (e.g., checks that committed files match decisions, verifies thread state in GitHub) and produces the attestation instead of the resolver itself.
Why: Self-attestation is the fundamental trust limitation. An independent observer would provide genuine separation of duties.

R14. Add human-in-the-loop gate for production deployment

Effort: Medium (workflow change)
Tickets: #132
What: Add a manual approval gate between the CI pipeline completing and actual deployment. The agentic pipeline produces evidence; a human reviews it before release.
Why: Currently on the happy path, AI authors code, AI reviews it, AI fixes it, AI ships it — no human touches it. Even with strong controls, regulators will want a human checkpoint.

R15. Implement prompt injection detection for diff content

Effort: Medium-Large (new control)
Tickets: #118
What: Scan diff content for adversarial patterns that could manipulate review agent behavior (e.g., "ignore all previous instructions" embedded in code comments).
Why: The diff is the primary input to all 10 review agents. Adversarial content in the diff could influence review outcomes.

---

Implementation Order

Week 1: R1 (generic→custom types) + R2 (commit timing) + R3 (hardcoded boolean)
Week 2: R4 (7 code bugs) + R6 (iteration limit) + R9 (shell injection)
Week 3: R5 (token scoping) + R7 (count reconciliation) + R8 (flow name validation)
Week 4: R10 (filter flag) + R11 (ticket integrity) + R12 (.kosli_ignore)
Future: R13 (independent observer) + R14 (human gate) + R15 (prompt injection)

---

Audit Complete ✅

All 31 attestation types have been systematically reviewed by 7 personas across 4 fill-out and 3 audit perspectives. The pipeline's evidence framework is fundamentally sound — the architecture of custom attestation types with JSON schemas and jq evaluator rules is the right approach. The priority remediation items above will close the gaps between the framework's potential and its current implementation.