|
| 1 | +--- |
| 2 | +phase: 07.1-llm-kg-builder-agent |
| 3 | +verified: 2026-02-08T12:40:00Z |
| 4 | +status: passed |
| 5 | +score: 8/8 must-haves verified |
| 6 | +must_haves: |
| 7 | + truths: |
| 8 | + - "KgEntity table has columns for aliases, description_brief, description_detailed, domains, source_finding_ids" |
| 9 | + - "KgRelationship table has columns for evidence_excerpt, source_finding_ids, temporal_context, corroboration_count, confidence" |
| 10 | + - "KgBuilderOutput Pydantic schema validates cleanly with Gemini structured output constraints" |
| 11 | + - "Existing KG API responses include the new fields when present" |
| 12 | + - "KG Builder agent receives all case_findings text + domain entity lists + case description as text-only input" |
| 13 | + - "KG Builder agent produces a KgBuilderOutput with curated entities (integer IDs) and semantic relationships" |
| 14 | + - "Old kg_entities and kg_relationships for the case are cleared before writing curated data" |
| 15 | + - "Pipeline replaces programmatic build_knowledge_graph() with LLM-based KG Builder invocation" |
| 16 | + artifacts: |
| 17 | + - path: "backend/alembic/versions/e4b2c1a37f90_evolve_kg_schema_for_llm_builder.py" |
| 18 | + provides: "Alembic migration adding new columns" |
| 19 | + - path: "backend/app/models/knowledge_graph.py" |
| 20 | + provides: "Updated ORM models with new Mapped columns" |
| 21 | + - path: "backend/app/schemas/kg_builder.py" |
| 22 | + provides: "KgBuilderOutput Pydantic schema for Gemini structured output" |
| 23 | + - path: "backend/app/schemas/knowledge_graph.py" |
| 24 | + provides: "Updated EntityResponse and RelationshipResponse" |
| 25 | + - path: "backend/app/agents/kg_builder.py" |
| 26 | + provides: "KgBuilderAgentRunner, input assembly, DB write, run_kg_builder" |
| 27 | + - path: "backend/app/agents/prompts/kg_builder.py" |
| 28 | + provides: "KG_BUILDER_SYSTEM_PROMPT" |
| 29 | + - path: "backend/app/agents/factory.py" |
| 30 | + provides: "create_kg_builder_agent static method" |
| 31 | + - path: "backend/app/services/pipeline.py" |
| 32 | + provides: "Stage 7 replaced with LLM KG Builder invocation" |
| 33 | + key_links: |
| 34 | + - from: "backend/app/agents/kg_builder.py" |
| 35 | + to: "backend/app/schemas/kg_builder.py" |
| 36 | + via: "KgBuilderOutput used as output_type" |
| 37 | + - from: "backend/app/agents/kg_builder.py" |
| 38 | + to: "backend/app/models/knowledge_graph.py" |
| 39 | + via: "writes KgEntity and KgRelationship ORM models" |
| 40 | + - from: "backend/app/services/pipeline.py" |
| 41 | + to: "backend/app/agents/kg_builder.py" |
| 42 | + via: "pipeline calls run_kg_builder()" |
| 43 | + - from: "backend/app/agents/factory.py" |
| 44 | + to: "backend/app/agents/prompts/kg_builder.py" |
| 45 | + via: "lazy import of KG_BUILDER_SYSTEM_PROMPT" |
| 46 | +gaps: [] |
| 47 | +--- |
| 48 | + |
| 49 | +# Phase 7.1: LLM-Based KG Builder Agent Verification Report |
| 50 | + |
| 51 | +**Phase Goal:** Replace the programmatic KG Builder with an LLM-based agent that reads ALL domain agent outputs holistically and produces a curated knowledge graph with deduplicated high-level entities and semantic relationships -- enabling cross-domain connections impossible with per-finding co-occurrence. |
| 52 | + |
| 53 | +**Verified:** 2026-02-08T12:40:00Z |
| 54 | +**Status:** PASSED |
| 55 | +**Re-verification:** No -- initial verification |
| 56 | + |
| 57 | +## Goal Achievement |
| 58 | + |
| 59 | +### Observable Truths |
| 60 | + |
| 61 | +| # | Truth | Status | Evidence | |
| 62 | +|---|-------|--------|----------| |
| 63 | +| 1 | KgEntity table has columns for aliases, description_brief, description_detailed, domains, source_finding_ids | VERIFIED | Alembic migration `e4b2c1a37f90` adds all 5 columns as nullable JSONB/String/Text. ORM model has matching `Mapped` fields at lines 79-103 of `knowledge_graph.py`. | |
| 64 | +| 2 | KgRelationship table has columns for evidence_excerpt, source_finding_ids, temporal_context, corroboration_count, confidence | VERIFIED | Same Alembic migration adds all 5 columns. ORM model has matching `Mapped` fields at lines 207-233 of `knowledge_graph.py`. corroboration_count defaults to 1, confidence defaults to 0.0. | |
| 65 | +| 3 | KgBuilderOutput Pydantic schema validates cleanly with Gemini structured output constraints | VERIFIED | `kg_builder.py` schema uses `list[MetadataEntry]` (not `dict`), no Union types, no discriminated unions. Integer ID cross-referencing pattern avoids name-matching. KgBuilderEntity has 11 fields, KgBuilderRelationship has 9 fields, KgBuilderOutput wraps both. | |
| 66 | +| 4 | Existing KG API responses include the new fields when present | VERIFIED | `EntityResponse` has aliases, description_brief, description_detailed, domains, source_finding_ids as Optional fields (lines 34-48 of `knowledge_graph.py` schemas). `RelationshipResponse` has evidence_excerpt, source_finding_ids, temporal_context, corroboration_count, confidence as Optional fields (lines 145-162). Both use `model_config = ConfigDict(from_attributes=True)` for ORM auto-mapping. Generated TypeScript types in `api.ts` include all new fields. | |
| 67 | +| 5 | KG Builder agent receives all case_findings text + domain entity lists + case description as text-only input | VERIFIED | `assemble_kg_builder_input()` at line 113 of `agents/kg_builder.py` queries CaseFinding rows (formatted with `[FINDING:{uuid}]` prefixes grouped by agent_type), extracts DomainEntity lists from domain_results (serialized as JSON), and queries Case.description. Returns 3-tuple of (findings_text, entities_json, case_description). `_prepare_content()` at line 55 builds a text-only `types.Content` from these three sources. | |
| 68 | +| 6 | KG Builder agent produces a KgBuilderOutput with curated entities (integer IDs) and semantic relationships | VERIFIED | `KgBuilderAgentRunner` subclasses `DomainAgentRunner[KgBuilderOutput]` (line 30). `_get_output_type()` returns `KgBuilderOutput`. Factory method at line 315 of `factory.py` creates LlmAgent with `output_schema=KgBuilderOutput`, `output_key="kg_builder_result"`, Pro model, high thinking. System prompt (126 lines) instructs 8+1 entity taxonomy, semantic relationship types, integer ID assignment, deduplication rules, evidence grounding requirements. | |
| 69 | +| 7 | Old kg_entities and kg_relationships for the case are cleared before writing curated data | VERIFIED | `write_kg_from_llm_output()` at line 207 of `agents/kg_builder.py` executes `delete(KgRelationship).where(...)` then `delete(KgEntity).where(...)` then `flush()` before inserting new entities. FK-safe order (relationships first). Entity degrees computed after write via `compute_entity_degrees()`. | |
| 70 | +| 8 | Pipeline replaces programmatic build_knowledge_graph() with LLM-based KG Builder invocation | VERIFIED | `pipeline.py` imports `run_kg_builder` from `app.agents.kg_builder` (line 156). Stage 7 (lines 968-1031) calls `run_kg_builder()` wrapped in try/except. Grep confirms `build_knowledge_graph` does NOT appear in pipeline.py. SSE events fire: `emit_agent_started` (line 986), `emit_agent_complete` (line 1003), `emit_agent_error` (line 1025). Failure is non-blocking -- sets `kg_entities_created=0`, continues pipeline. | |
| 71 | + |
| 72 | +**Score:** 8/8 truths verified |
| 73 | + |
| 74 | +### Required Artifacts |
| 75 | + |
| 76 | +| Artifact | Expected | Status | Details | |
| 77 | +|----------|----------|--------|---------| |
| 78 | +| `backend/alembic/versions/e4b2c1a37f90_evolve_kg_schema_for_llm_builder.py` | Migration adding 10 nullable columns | VERIFIED | 154 lines. Adds 5 columns to kg_entities (aliases, description_brief, description_detailed, domains, source_finding_ids) and 5 to kg_relationships (evidence_excerpt, source_finding_ids, temporal_context, corroboration_count, confidence). Proper upgrade/downgrade. | |
| 79 | +| `backend/app/models/knowledge_graph.py` | Updated ORM models | VERIFIED | 245 lines. KgEntity has all 5 new Mapped fields (lines 79-103). KgRelationship has all 5 new Mapped fields (lines 207-233). Docstrings updated to reference "LLM-based KG Builder agent" and "semantic relationship extraction". | |
| 80 | +| `backend/app/schemas/kg_builder.py` | KgBuilderOutput Pydantic schema | VERIFIED | 141 lines. KgBuilderEntity (11 fields including integer id, aliases, description_brief/detailed, domains, source_finding_ids, properties via MetadataEntry). KgBuilderRelationship (9 fields including evidence_excerpt, temporal_context, confidence). KgBuilderOutput wrapper. ABOUTME comments present. | |
| 81 | +| `backend/app/schemas/knowledge_graph.py` | Updated API response schemas | VERIFIED | 221 lines. EntityResponse has 5 new Optional fields (lines 34-48). RelationshipResponse has 5 new Optional fields (lines 145-162). All use `from_attributes=True`. | |
| 82 | +| `backend/app/agents/kg_builder.py` | KgBuilderAgentRunner + run_kg_builder + input assembly + DB write | VERIFIED | 423 lines. KgBuilderAgentRunner subclass (lines 30-105). assemble_kg_builder_input() (lines 113-199). write_kg_from_llm_output() (lines 207-350). run_kg_builder() top-level (lines 358-423). ABOUTME comments present. | |
| 83 | +| `backend/app/agents/prompts/kg_builder.py` | KG_BUILDER_SYSTEM_PROMPT | VERIFIED | 126 lines. 8+1 entity taxonomy table, entity instructions (7 items), relationship instructions (7 items), what-not-to-do section, input/output format. Instructs semantic labels, evidence grounding, deduplication, cross-domain connections. ABOUTME comments present. | |
| 84 | +| `backend/app/agents/factory.py` | create_kg_builder_agent() | VERIFIED | Method at line 315 (33 lines). Uses MODEL_PRO, thinking_level "high", lazy imports KG_BUILDER_SYSTEM_PROMPT and KgBuilderOutput. No generate_content_config (text-only). Follows exact pattern of other factory methods. | |
| 85 | +| `backend/app/services/kg_builder.py` | Old functions marked deprecated | VERIFIED | 435 lines. `extract_entities_from_output` (line 30), `build_relationships_from_findings` (line 110), `deduplicate_entities` (line 201), `build_knowledge_graph` (line 359) all prefixed with "DEPRECATED: Replaced by LLM-based KG Builder agent (Phase 7.1). Kept for reference." `normalize_entity_name()` and `compute_entity_degrees()` preserved for reuse. | |
| 86 | +| `backend/app/services/pipeline.py` | Stage 7 replaced | VERIFIED | 1147 lines. Stage 7 at lines 968-1031 calls `run_kg_builder()`. try/except wraps invocation. SSE lifecycle events (started/complete/error). Strategy output appended to domain_results before KG Builder (lines 975-983). | |
| 87 | +| `packages/types/src/generated/api.ts` | Frontend types include new fields | VERIFIED | TypeScript types contain aliases, description_brief, description_detailed, domains, source_finding_ids (entity), evidence_excerpt, source_finding_ids, temporal_context, corroboration_count (relationship). | |
| 88 | + |
| 89 | +### Key Link Verification |
| 90 | + |
| 91 | +| From | To | Via | Status | Details | |
| 92 | +|------|----|-----|--------|---------| |
| 93 | +| `agents/kg_builder.py` | `schemas/kg_builder.py` | KgBuilderOutput used as output_type | VERIFIED | Line 42: `_get_output_type() returns KgBuilderOutput`. Line 20: `from app.schemas.kg_builder import KgBuilderOutput`. | |
| 94 | +| `agents/kg_builder.py` | `models/knowledge_graph.py` | writes KgEntity/KgRelationship ORM objects | VERIFIED | Line 19: `from app.models.knowledge_graph import KgEntity, KgRelationship`. write_kg_from_llm_output() creates KgEntity instances (line 251) and KgRelationship instances (line 305). | |
| 95 | +| `services/pipeline.py` | `agents/kg_builder.py` | calls run_kg_builder() | VERIFIED | Line 156: `from app.agents.kg_builder import run_kg_builder`. Line 995: `await run_kg_builder(...)`. Old `build_knowledge_graph` NOT referenced (grep returns no matches). | |
| 96 | +| `agents/factory.py` | `agents/prompts/kg_builder.py` | lazy import of KG_BUILDER_SYSTEM_PROMPT | VERIFIED | Line 335: `from app.agents.prompts.kg_builder import KG_BUILDER_SYSTEM_PROMPT`. Used in `_create_llm_agent()` call at line 341. | |
| 97 | +| `schemas/knowledge_graph.py` | `models/knowledge_graph.py` | from_attributes=True reads new ORM columns | VERIFIED | EntityResponse line 68: `model_config = ConfigDict(from_attributes=True)`. RelationshipResponse line 165: same. Field names match ORM column names exactly. | |
| 98 | +| `agents/kg_builder.py` | `services/kg_builder.py` | reuses normalize_entity_name and compute_entity_degrees | VERIFIED | Line 228: `from app.services.kg_builder import compute_entity_degrees, normalize_entity_name`. Used in write_kg_from_llm_output() at line 254 (normalize_entity_name) and line 337 (compute_entity_degrees). | |
| 99 | +| `agents/kg_builder.py` | `agents/domain_agent_runner.py` | subclasses DomainAgentRunner | VERIFIED | Line 30: `class KgBuilderAgentRunner(DomainAgentRunner[KgBuilderOutput])`. DomainAgentRunner.run() accepts `**kwargs` (line 194) enabling findings_text/entities_json/case_description passthrough. | |
| 100 | +| KG API endpoints | new KG data | GraphResponse uses from_attributes | VERIFIED | `api/knowledge_graph.py` line 98: `EntityResponse.model_validate(e)` -- automatically picks up all new ORM columns (aliases, description_brief, etc.) via from_attributes. No API route changes needed. | |
| 101 | + |
| 102 | +### Requirements Coverage |
| 103 | + |
| 104 | +| Requirement | Status | Blocking Issue | |
| 105 | +|-------------|--------|----------------| |
| 106 | +| REQ-AGENT-009 (revised -- LLM-based KG Builder) | SATISFIED | None. Agent reads all findings holistically, produces curated entities with semantic relationships. | |
| 107 | + |
| 108 | +### Anti-Patterns Found |
| 109 | + |
| 110 | +| File | Line | Pattern | Severity | Impact | |
| 111 | +|------|------|---------|----------|--------| |
| 112 | +| None found | -- | -- | -- | No TODO, FIXME, placeholder, or stub patterns detected in any of the 10 artifacts. | |
| 113 | + |
| 114 | +### Human Verification Required |
| 115 | + |
| 116 | +#### 1. Graph Quality Test |
| 117 | +**Test:** Run the full pipeline on a multi-file case (e.g., financial + legal documents) and inspect the resulting knowledge graph via `GET /api/cases/:caseId/graph`. |
| 118 | +**Expected:** ~15-30 curated entities (not 50+ noisy ones), relationships with semantic labels like "CEO of" or "transferred_funds_to" (not "co-occurrence"), entities with populated aliases arrays, relationships with evidence_excerpt fields containing actual quotes. |
| 119 | +**Why human:** Graph quality depends on Gemini Pro's response to the prompt. Structural verification confirms the pipeline exists and is wired correctly, but cannot predict LLM output quality. |
| 120 | + |
| 121 | +#### 2. Cross-Domain Deduplication Test |
| 122 | +**Test:** Upload files that mention the same person across financial and legal domains (e.g., "J. Smith" in financial docs, "John Smith" in legal docs). Check if the KG Builder merges them into one entity with both as aliases. |
| 123 | +**Expected:** Single entity with aliases `["J. Smith", "John Smith"]` and domains `["financial", "legal"]`. |
| 124 | +**Why human:** Deduplication quality depends on the LLM's ability to recognize the same person across different name variants and contexts. |
| 125 | + |
| 126 | +#### 3. No Standalone Timestamp/Amount Nodes Test |
| 127 | +**Test:** Inspect the graph for entities of type that should be metadata (e.g., "$5,000", "2023-01-15", "Model XYZ"). These should NOT be standalone nodes. |
| 128 | +**Expected:** Monetary amounts, timestamps, and physical objects appear as properties on entities/relationships, not as standalone graph nodes. |
| 129 | +**Why human:** The prompt instructs this, but LLM compliance varies. |
| 130 | + |
| 131 | +#### 4. Pipeline Graceful Failure Test |
| 132 | +**Test:** Trigger a KG Builder failure (e.g., by temporarily breaking the Gemini API key) and verify the pipeline continues past Stage 7. |
| 133 | +**Expected:** Error logged, SSE error event emitted, pipeline continues to Stage 7b and final steps, case status becomes READY (not ERROR). |
| 134 | +**Why human:** Requires runtime execution to verify error handling path. |
| 135 | + |
| 136 | +### Gaps Summary |
| 137 | + |
| 138 | +No gaps found. All 8 must-haves are verified at all three levels (exists, substantive, wired): |
| 139 | + |
| 140 | +1. **DB Schema Evolution** -- Alembic migration adds 10 nullable columns, ORM models have matching Mapped fields, API response schemas expose new fields as Optional. |
| 141 | +2. **Pydantic Output Schema** -- KgBuilderOutput with integer ID cross-referencing, Gemini-compatible (no dict types), 3 models totaling 141 lines. |
| 142 | +3. **KG Builder Agent** -- KgBuilderAgentRunner subclass with text-only content preparation, input assembly from case_findings + domain entities + case description, DB write with clear-and-rebuild strategy, lenient parsing, entity degree computation. |
| 143 | +4. **System Prompt** -- 126-line prompt with 8+1 entity taxonomy, semantic relationship instructions, deduplication rules, evidence grounding requirements, anti-patterns ("do NOT create standalone timestamp nodes"). |
| 144 | +5. **Factory Method** -- create_kg_builder_agent() in AgentFactory, Pro model, high thinking, text-only (no media resolution). |
| 145 | +6. **Pipeline Wiring** -- Stage 7 calls run_kg_builder() instead of build_knowledge_graph(), wrapped in try/except for graceful failure, SSE lifecycle events for started/complete/error. |
| 146 | +7. **Old Code Preserved** -- 4 programmatic functions marked deprecated, normalize_entity_name() and compute_entity_degrees() reused. |
| 147 | +8. **API Compatibility** -- Existing KG API endpoints return curated data without any route changes (from_attributes auto-maps new ORM columns). |
| 148 | + |
| 149 | +--- |
| 150 | + |
| 151 | +_Verified: 2026-02-08T12:40:00Z_ |
| 152 | +_Verifier: Claude (gsd-verifier)_ |
0 commit comments