docs: update architecture and API references

Author: hammertoe

docs/CODE_MAP_AND_REVIEW.md

@@ -26,6 +26,78 @@ YuhHearDem3 is a parliamentary transcription and knowledge graph system that pro
 └─────────────────────────────────────────────────────────────────────────────┘
 ```
 
+## Code Flow Diagram (Mermaid)
+
+```mermaid
+flowchart LR
+  subgraph Sources
+    YT[YouTube or GCS video]
+    OPDF[Order paper PDF]
+    BillsSite[Parliament bills site]
+  end
+
+  subgraph Transcription
+    Transcribe[transcribe.py]
+    JSONOut[transcription_output.json]
+  end
+
+  subgraph TranscriptIngest
+    IngestScript[scripts/ingest_transcript_json.py]
+    Ingestor[lib/transcripts/ingestor.py]
+  end
+
+  subgraph OrderPapers
+    OPIngest[scripts/ingest_order_paper_pdf.py]
+    OPParser[lib/order_papers/*.py]
+  end
+
+  subgraph Bills
+    BillIngest[scripts/ingest_bills.py]
+    BillScraper[lib/scraping/bill_scraper.py]
+    BillProcessor[lib/processors/bill_ingestor.py]
+  end
+
+  subgraph KGExtraction
+    KGVideo[scripts/kg_extract_from_video.py]
+    KGBills[scripts/kg_extract_from_bills.py]
+    WindowBuilder[lib/knowledge_graph/window_builder.py]
+    BillWindowBuilder[lib/knowledge_graph/bill_window_builder.py]
+    Extractor[lib/knowledge_graph/oss_kg_extractor.py + kg_extractor.py]
+    KGStore[lib/knowledge_graph/kg_store.py]
+  end
+
+  subgraph Storage[(PostgreSQL + pgvector)]
+    Tables[Transcript + search + KG tables]
+  end
+
+  subgraph SearchAPI
+    API[api/search_api.py]
+    ChatAgent[lib/chat_agent_v2.py]
+    AgentLoop[lib/kg_agent_loop.py]
+    HybridRAG[lib/kg_hybrid_graph_rag.py]
+    AdvSearch[lib/advanced_search_features.py]
+  end
+
+  subgraph Frontend
+    UI[frontend/src (Vite + React)]
+  end
+
+  YT --> Transcribe --> JSONOut --> IngestScript --> Ingestor --> Tables
+  OPDF --> OPIngest --> OPParser --> Tables
+  BillsSite --> BillIngest --> BillScraper --> BillProcessor --> Tables
+
+  Tables --> WindowBuilder --> KGVideo
+  Tables --> BillWindowBuilder --> KGBills
+  KGVideo --> Extractor --> KGStore --> Tables
+  KGBills --> Extractor --> KGStore
+
+  Tables --> API
+  API --> ChatAgent --> AgentLoop --> HybridRAG --> Tables
+  API --> AdvSearch --> Tables
+  UI --> API
+  API --> UI
+```
+
 ## Code Map
 
 ### Entry Points
@@ -50,12 +122,15 @@ YuhHearDem3 is a parliamentary transcription and knowledge graph system that pro
 
 | File | Lines | Purpose |
 |------|-------|---------|
+| `oss_kg_extractor.py` | ~800 | OSS KG extraction (two-pass) |
 | `oss_two_pass.py` | 677 | OSS two-pass entity extraction |
 | `window_builder.py` | 287 | Window-based processing for transcripts |
+| `bill_window_builder.py` | ~200 | Bill excerpt window construction |
 | `kg_store.py` | ~350 | KG storage operations |
 | `kg_extractor.py` | ~550 | Main KG extraction logic |
 | `base_kg_seeder.py` | ~300 | Base KG seeding |
 | `model_compare.py` | ~300 | Model comparison utilities |
+| `window_benchmark.py` | ~160 | Window performance benchmarks |
 
 #### Order Papers (`lib/order_papers/`)
 
@@ -74,6 +149,12 @@ YuhHearDem3 is a parliamentary transcription and knowledge graph system that pro
 |------|-------|---------|
 | `ingestor.py` | 433 | Transcript ingestion |
 
+#### Embeddings (`lib/embeddings/`)
+
+| File | Lines | Purpose |
+|------|-------|---------|
+| `google_client.py` | ~200 | Embedding generation client |
+
 #### Database (`lib/db/`)
 
 | File | Lines | Purpose |
@@ -101,18 +182,26 @@ YuhHearDem3 is a parliamentary transcription and knowledge graph system that pro
 | File | Lines | Purpose |
 |------|-------|---------|
 | `config.py` | 85 | Configuration management |
-| `roles.py` | ~50 | Role utilities |
+
+#### Utilities (`lib/`)
+
+| File | Lines | Purpose |
+|------|-------|---------|
 | `id_generators.py` | ~100 | ID generation utilities |
+| `roles.py` | ~120 | Speaker role normalization utilities |
 
 ### Scripts (`scripts/`)
 
 | File | Purpose |
 |------|---------|
 | `kg_extract_from_video.py` | Extract KG from video |
+| `kg_extract_from_bills.py` | Extract KG from bill excerpts |
 | `cron_transcription.py` | Automated transcription jobs |
 | `migrate_chat_schema.py` | Chat schema migration |
 | `clear_kg.py` | Clear KG tables |
 | `ingest_order_paper_pdf.py` | Ingest order paper PDFs |
+| `ingest_transcript_json.py` | Ingest transcript JSON into Postgres |
+| `ingest_bills.py` | Scrape/process bills and ingest |
 | `ingest_knowledge_graph.py` | Ingest KG data |
 | `list_channel_videos.py` | List channel videos |
 | `match_order_papers_to_videos.py` | Match papers to videos |

docs/COMPLETE_GUIDE.md

@@ -33,6 +33,8 @@ A hybrid vector/graph search and conversational AI system for Barbados Parliamen
 └─────────────────────────────────────────────────────────────────────────────┘
 ```
 
+The frontend is served by FastAPI from `frontend/dist` and talks to the same API origin.
+
 ---
 
 ## Core Modules
@@ -109,12 +111,26 @@ CHAT_TRACE=1 python -m uvicorn api.search_api:app --reload
 
 ```
 Video URL → yt-dlp metadata → Gemini API → Segment transcription → Speaker normalization → JSON output
+JSON output → scripts/ingest_transcript_json.py → Transcript tables (videos/paragraphs/sentences/entities)
+```
+
+### Order Paper Flow
+
+```
+Order paper PDF → scripts/ingest_order_paper_pdf.py → order_papers/order_paper_items → context + role seeding
+```
+
+### Bill Ingestion Flow
+
+```
+Bill site → scripts/ingest_bills.py → bills + bill_excerpts (embeddings)
 ```
 
 ### Knowledge Graph Flow
 
 ```
 Transcript → Window Builder (30 utterances, stride 18) → LLM extraction → Canonicalization → KG Store → PostgreSQL
+Bill excerpts → BillWindowBuilder → LLM extraction → Canonicalization → KG Store → PostgreSQL
 ```
 
 ### Chat Flow
@@ -155,14 +171,16 @@ User Query → Embedding → Vector Search → Graph Expansion → LLM Synthesis
 | Method | Path | Description |
 |--------|------|-------------|
 | POST | `/search` | Hybrid search (vector + graph + BM25) |
-| POST | `/chat` | Conversational AI with citations |
-| GET | `/chat/threads` | List chat threads |
-| POST | `/chat/threads` | Create new thread |
-| GET | `/chat/threads/{id}` | Get thread messages |
-| POST | `/chat/threads/{id}` | Add message to thread |
-| GET | `/graph` | Graph data for entity |
+| POST | `/search/temporal` | Search with date/speaker/entity filters |
+| GET | `/search/trends` | Trend analysis for entities |
 | GET | `/speakers` | List all speakers |
-| GET | `/speakers/{id}` | Speaker details |
+| GET | `/speakers/{speaker_id}` | Speaker details |
+| GET | `/videos/{youtube_video_id}/speakers/{speaker_id}/roles` | Speaker roles for a video |
+| POST | `/chat/threads` | Create new thread |
+| POST | `/chat/threads/{thread_id}/messages` | Add message to thread |
+| GET | `/chat/threads/{thread_id}/messages/stream` | Stream message response (SSE) |
+| GET | `/health` | Health check |
+| GET | `/api` | API metadata |
 
 ---
 
@@ -171,11 +189,14 @@ User Query → Embedding → Vector Search → Graph Expansion → LLM Synthesis
 | Script | Purpose |
 |--------|---------|
 | `transcribe.py` | Main video transcription |
+| `scripts/ingest_transcript_json.py` | Ingest transcript JSON into Postgres |
 | `scripts/kg_extract_from_video.py` | Extract KG from video |
+| `scripts/kg_extract_from_bills.py` | Extract KG from bill excerpts |
 | `scripts/cron_transcription.py` | Automated transcription jobs |
 | `scripts/migrate_chat_schema.py` | Chat schema migration |
 | `scripts/clear_kg.py` | Clear KG tables |
 | `scripts/ingest_order_paper_pdf.py` | Ingest order paper PDFs |
+| `scripts/ingest_bills.py` | Scrape/process bills and ingest |
 | `scripts/list_channel_videos.py` | List channel videos |
 
 ---
@@ -195,9 +216,12 @@ User Query → Embedding → Vector Search → Graph Expansion → LLM Synthesis
 **transcribe.py:**
 
 - `--order-file`: Path to order paper file
+- `--order-paper-id`: Order paper ID from database
 - `--segment-minutes`: Segment duration (default: 30)
+- `--overlap-minutes`: Segment overlap (default: 1)
 - `--start-minutes`: Start position (default: 0)
 - `--max-segments`: Limit segments processed
+- `--video`: YouTube ID/URL or gs:// URI
 
 **kg_extract_from_video.py:**
 

docs/QUICK_REFERENCE.md

@@ -24,6 +24,9 @@ python transcribe.py --order-file order.txt --max-segments 2
 # Extract KG from video
 python scripts/kg_extract_from_video.py --youtube-video-id "VIDEO_ID"
 
+# Extract KG from bill excerpts
+python scripts/kg_extract_from_bills.py --max-bills 10
+
 # With custom window parameters
 python scripts/kg_extract_from_video.py --youtube-video-id "VIDEO_ID" --window-size 15 --stride 10
 
@@ -71,6 +74,12 @@ python scripts/migrate_chat_schema.py
 
 # Backfill speaker roles
 python scripts/backfill_speaker_video_roles.py
+
+# Ingest transcript JSON into Postgres
+python scripts/ingest_transcript_json.py --transcript-file transcription_output.json --youtube-video-id "VIDEO_ID"
+
+# Ingest bills into Postgres
+python scripts/ingest_bills.py --scrape
 ```
 
 ### Order Papers
@@ -111,14 +120,16 @@ mypy lib/
 | Method | Endpoint | Description |
 |--------|----------|-------------|
 | POST | `/search` | Hybrid search |
-| POST | `/chat` | Conversational AI |
-| GET | `/chat/threads` | List threads |
-| POST | `/chat/threads` | Create thread |
-| GET | `/chat/threads/{id}` | Get thread |
-| POST | `/chat/threads/{id}` | Send message |
-| GET | `/graph` | Graph data |
+| POST | `/search/temporal` | Search with date/speaker/entity filters |
+| GET | `/search/trends` | Trend analysis for entities |
 | GET | `/speakers` | List speakers |
-| GET | `/speakers/{id}` | Speaker details |
+| GET | `/speakers/{speaker_id}` | Speaker details |
+| GET | `/videos/{youtube_video_id}/speakers/{speaker_id}/roles` | Speaker roles for a video |
+| POST | `/chat/threads` | Create thread |
+| POST | `/chat/threads/{thread_id}/messages` | Send message |
+| GET | `/chat/threads/{thread_id}/messages/stream` | Stream message response |
+| GET | `/health` | Health check |
+| GET | `/api` | API metadata |
 
 ## Environment Variables
 
@@ -156,10 +167,13 @@ mypy lib/
 | Option | Default | Description |
 |--------|---------|-------------|
 | `--order-file | Path` | Required to order file |
+| `--order-paper-id` | None | Order paper ID from database |
 | `--segment-minutes` | 30 | Segment duration |
+| `--overlap-minutes` | 1 | Segment overlap |
 | `--start-minutes` | 0 | Start position |
 | `--max-segments` | None | Limit segments |
 | `--output-file` | Varies | Output file path |
+| `--video` | None | YouTube ID/URL or gs:// URI |
 
 ### kg_extract_from_video.py
 

docs/README.md

@@ -27,6 +27,12 @@ A comprehensive parliamentary transcription and search system that processes vid
 - **Follow-Up Suggestions**: Generates contextual follow-up questions
 - **Full Citation Tracing**: Every answer grounded in transcript evidence
 
+### Frontend UI
+
+- **React + Vite**: Single-page app served from `frontend/dist`
+- **Streaming Chat**: SSE-based progress updates
+- **Graph View**: Explore entity connections visually
+
 ### Search System
 
 - **Hybrid Search**: Combines vector similarity, BM25 full-text, and graph traversal
@@ -56,6 +62,8 @@ A comprehensive parliamentary transcription and search system that processes vid
 └─────────────────────────────────────────────────────────────────────────┘
 ```
 
+The frontend is served by FastAPI from `frontend/dist` and talks to the same API origin.
+
 ## Quick Start
 
 ### Prerequisites
@@ -94,6 +102,12 @@ python transcribe.py --order-file order.txt --segment-minutes 30
 python scripts/kg_extract_from_video.py --youtube-video-id "VIDEO_ID"
 ```
 
+### Ingest Transcript JSON
+
+```bash
+python scripts/ingest_transcript_json.py --transcript-file transcription_output.json --youtube-video-id "VIDEO_ID"
+```
+
 ### Start Chat API
 
 ```bash
@@ -111,6 +125,7 @@ YuhHearDem3/
 │   ├── kg_agent_loop.py           # KG-powered agent loop
 │   ├── kg_hybrid_graph_rag.py     # Hybrid Graph-RAG retrieval
 │   ├── advanced_search_features.py # Temporal search, trends, graph queries
+│   ├── embeddings/                # Embedding clients
 │   ├── knowledge_graph/
 │   │   ├── oss_two_pass.py       # OSS two-pass extraction
 │   │   ├── window_builder.py      # Window-based processing
@@ -127,6 +142,7 @@ YuhHearDem3/
 │   ├── cron_transcription.py      # Automated transcription
 │   ├── migrate_chat_schema.py     # Chat schema migration
 │   └── clear_kg.py                # Clear KG tables
+├── frontend/                       # React frontend (Vite)
 ├── tests/                          # Unit tests
 └── docs/                           # Documentation
 ```
@@ -139,6 +155,8 @@ YuhHearDem3/
 | [QUICK_REFERENCE.md](QUICK_REFERENCE.md) | Command quick reference |
 | [CHAT_TRACE.md](CHAT_TRACE.md) | Debug tracing documentation |
 | [DATE_NORMALIZATION.md](DATE_NORMALIZATION.md) | Date handling |
+| [CODE_MAP_AND_REVIEW.md](CODE_MAP_AND_REVIEW.md) | Code map and flow diagram |
+| [README_SEARCH_SYSTEM.md](README_SEARCH_SYSTEM.md) | Search system details |
 
 ## Technology Stack
 

docs/README_SEARCH_SYSTEM.md

@@ -137,15 +137,24 @@ LIMIT 50;
 }
 ```
 
-### POST /chat
+### POST /search/temporal
 
 ```json
 {
-  "thread_id": "uuid",
-  "message": "What did they say about healthcare?"
+  "query": "healthcare reform",
+  "limit": 20,
+  "alpha": 0.6,
+  "start_date": "2024-01-01",
+  "end_date": "2024-12-31",
+  "speaker_id": "s_john_doe_1",
+  "entity_type": "schema:Legislation"
 }
 ```
 
+### GET /search/trends
+
+Query params: `entity_id`, `days`, `window_size`
+
 ## Performance
 
 | Operation | Typical Latency |