LDRS v3 — Hybrid RAG Deep Agent System

Living Document RAG System, version 3 — a 6-stage retrieval-augmented generation pipeline that combines hierarchical grep search, pgvector semantic similarity, BM25 fusion ranking, and an agentic reasoning loop with grounding verification. Designed for multi-document technical knowledge bases with full Nepali/Devanagari Unicode support.

Python 3.12+   |   PostgreSQL 16 + pgvector   |   OpenAI-compatible API

---

Table of Contents

1. Architecture Overview
2. Pipeline Stages
3. Quick Start
4. Configuration Reference
5. API Reference
6. Streamlit UI
7. Module Reference
8. Indexing & File Watcher
9. Virtual Filesystem (VFS)
10. Agent Tools & Function Calling
11. Grounding Verification
12. Monitoring & LangSmith
13. Citation Format
14. Testing
15. Project Structure
16. Troubleshooting

---

Architecture Overview

LDRS v3 processes user queries through a deterministic 6-stage pipeline where each stage has a well-defined input/output contract. Three stages involve LLM calls (Intent Classification, Agent Loop, Grounding); the remaining stages are deterministic.

                          ┌──────────────────────────┐
                          │   Source .md Files        │
                          │   (docs/ directory)       │
                          └────────────┬─────────────┘
                                       │
                          ┌────────────▼─────────────┐
                          │   File Watcher (Stage 0)  │
                          │   watchdog + debounce     │
                          └────────────┬─────────────┘
                                       │
                          ┌────────────▼─────────────┐
                          │   PageIndex (md_to_tree)  │
                          │   .md → structure JSON    │
                          └────────────┬─────────────┘
                                       │
                     ┌─────────────────┼─────────────────┐
                     │                 │                  │
            ┌────────▼───────┐ ┌──────▼────────┐ ┌──────▼───────┐
            │ Structure JSON │ │  Embed into   │ │  Registry    │
            │ (results/)     │ │  pgvector     │ │  (JSON)      │
            └────────────────┘ └───────────────┘ └──────────────┘


  User Query ──────────────────────────────────────────────────────────
                     │
            ┌────────▼────────────────────────────────────────────┐
            │  Stage 1: Intent Classification (LLM)               │
            │  → intent_type, selected_files, query_variants,     │
            │    pattern_hints, needs_db, likely_multihop          │
            └────────┬────────────────────────────────────────────┘
                     │
            ┌────────▼────────────────────────────────────────────┐
            │  Stage 2: Parallel Retrieval                        │
            │  ┌──────────────┐    ┌──────────────────────┐       │
            │  │  TreeGrep    │    │  pgvector Similarity  │       │
            │  │  (pattern    │ ∥  │  (query_variants →    │       │
            │  │   hints)     │    │   cosine search)      │       │
            │  └──────┬───────┘    └──────────┬───────────┘       │
            │         └──────────┬────────────┘                   │
            │                    ▼                                │
            │         Section Pool (deduplicated)                 │
            └────────┬────────────────────────────────────────────┘
                     │
            ┌────────▼────────────────────────────────────────────┐
            │  Stage 3: BM25 Fusion Ranking                       │
            │  score = (w_bm25 × BM25) + (w_vec × vector)        │
            │        + (w_grep × grep_density)                    │
            │  × recency_factor × tag_boost                       │
            │  Weights shift by intent_type                       │
            └────────┬────────────────────────────────────────────┘
                     │
            ┌────────▼────────────────────────────────────────────┐
            │  Stage 4: VFS Population                            │
            │  /sessions/{id}/manifest.json                       │
            │  /sessions/{id}/retrieved/rank1__doc__section.md     │
            │  /sessions/{id}/conversation/                       │
            │  /sessions/{id}/working/scratchpad.md                │
            └────────┬────────────────────────────────────────────┘
                     │
            ┌────────▼────────────────────────────────────────────┐
            │  Stage 5: Agent Loop (LLM + Function Calling)       │
            │  Read manifest → self-sufficiency check →            │
            │  read_section / fetch_section / scratchpad →          │
            │  cited answer synthesis                              │
            └────────┬────────────────────────────────────────────┘
                     │
            ┌────────▼────────────────────────────────────────────┐
            │  Stage 6: Grounding Verification (LLM)              │
            │  Extract claims → entailment check per claim →       │
            │  flag unsupported → caveat insertion →                │
            │  optional re-grounding loop                         │
            └────────┬────────────────────────────────────────────┘
                     │
                     ▼
              Final Verified Answer

Key Design Decisions

• OpenAI function calling is used directly (not through a framework) for the agent loop, giving fine-grained control over tool execution, message history, and forced synthesis on iteration limits.

• Dual retrieval (TreeGrep + pgvector) ensures both keyword-exact and semantic matches are captured. BM25 fusion ranking merges these signals with intent-aware weight presets.

• VFS-per-session gives the agent a self-contained filesystem with ranked sections, conversation context, and a scratchpad. The agent navigates via manifest.json and reads sections selectively.

• Grounding verification runs after every agent answer. Each cited claim is checked via LLM entailment against its source section. Unsupported claims get caveats; high flag ratios trigger re-grounding.

• NFC normalization is applied at every text boundary for safe handling of Nepali/Devanagari characters.

---

Pipeline Stages

Stage 0: File Watcher

Monitors directories for .md file changes using the watchdog library. On create/modify, the file is re-indexed (parsed, embedded, registered). On delete, embeddings and registry entries are removed. A configurable debounce (default 2 seconds) prevents rapid successive re-indexing.

Stage 1: Intent Classification

A single LLM call receives the user query and the compact registry JSON. It outputs a structured JSON with:

Field	Description
intent_type	One of: exact, conceptual, comparative, multihop, db_query, hybrid
selected_files	Files likely to contain the answer, with confidence scores
query_variants	2-4 rephrased queries for diverse retrieval
pattern_hints	{literals, phrases, prefix_wildcards} for TreeGrep
needs_db	Whether database access is needed
likely_multihop	Whether multi-hop reasoning is required

Fast paths: empty registry returns defaults without an LLM call.

Stage 2: Parallel Retrieval

Two retrieval methods run concurrently:

1. TreeGrep — searches structure JSONs using pattern_hints from Stage 1. Three-tier matching: title (3.0), summary (2.0), body (1.0). Supports exact substring and word-level matching with stop word filtering and configurable minimum match ratio (0.3).

2. pgvector — embeds each query_variant and runs cosine similarity search against the sections table. Multi-query results are deduplicated by (doc_name, section_id), keeping the highest similarity score.

Results are merged into a unified SectionCandidate pool with combined grep and vector signals.

Stage 3: BM25 Fusion Ranking

Scores each candidate using a weighted fusion:

raw_score = (w_bm25 × BM25_norm) + (w_vector × similarity) + (w_grep × grep_density)
final_score = raw_score × recency_factor × tag_boost

Weight presets by intent type:

Intent	BM25	Vector	Grep
exact	0.30	0.35	0.35
conceptual	0.25	0.55	0.20
comparative	0.35	0.40	0.25
multihop	0.35	0.40	0.25
db_query	0.30	0.40	0.30
hybrid	0.35	0.40	0.25

Metadata boosts:

• recency_factor: exponential decay with 365-day half-life, range [0.5, 1.0]
• tag_boost: fraction of query tokens matching registry tags, range [1.0, 1.5]

For comparative intent, a round-robin interleave ensures balanced multi-file coverage.

Stage 4: VFS Population

Creates a per-session directory with:

• manifest.json — ranked section list with metadata and scores
• retrieved/ — individual .md files per ranked section
• conversation/ — history summary and recent turns
• db_context/ — optional database query results
• working/scratchpad.md — agent working memory

The manifest is the agent's primary navigation interface.

Stage 5: Agent Loop

An iterative OpenAI function-calling loop:

1. The agent receives the query, intent type, and manifest
2. It calls tools (read_section, fetch_section, write_scratchpad, etc.)
3. Each iteration is an LLM call with tool_choice="auto"
4. When the LLM produces a text response without tool calls, the loop ends
5. If max iterations are reached, a forced synthesis prompt is sent

Tool definitions are provided in OpenAI function-calling format. Only sections actually read via read_section or fetch_section may be cited.

Stage 6: Grounding Verification

Post-answer verification of every cited claim:

1. Extract (claim, citation) pairs from the answer text
2. Locate cited section content in the VFS
3. LLM entailment check: "Does this section support this claim?"
4. Flag unsupported claims; insert caveat text
5. If flag ratio > 0.4, set re_grounded=True; the pipeline re-runs Stage 5 with grounding feedback appended to the query

Flagged claims are logged to results/hallucination_log.jsonl.

---

Quick Start

Prerequisites

• Python 3.12+
• Docker (for PostgreSQL + pgvector)
• An OpenAI-compatible API endpoint (local or remote)

1. Clone and install

git clone <repo-url> ldrs_v3
cd ldrs_v3
pip install -r requirements.txt

Or install as a package:

pip install -e ".[dev]"

2. Start PostgreSQL + pgvector

docker compose up -d

This starts pgvector/pgvector:pg16 on port 5432 and runs scripts/init_db.sql to create the sections and hallucination_log tables with the pgvector extension.

3. Configure environment

cp .env.example .env
# Edit .env with your API key, base URL, and other settings

4. Index documents

Place your .md source files in docs/, then index:

# Via API (start server first)
uvicorn api.server:app --host 0.0.0.0 --port 8001

# Then index via HTTP
curl -X POST http://localhost:8001/index-directory

Or programmatically:

import asyncio
from agent.config import AgentConfig
from agent.pipeline import Pipeline

async def main():
    pipeline = Pipeline(AgentConfig())
    await pipeline.startup()
    results = await pipeline.index_directory("docs/")
    for r in results:
        print(f"{r.doc_name}: {r.node_count} nodes, {r.embedded_count} embedded")
    await pipeline.shutdown()

asyncio.run(main())

5. Query

curl -X POST http://localhost:8001/query \
  -H "Content-Type: application/json" \
  -d '{"query": "How does OAuth2 token refresh work?"}'

6. Launch the UI

streamlit run ui/streamlit_app.py

---

Configuration Reference

All configuration is centralized in AgentConfig (agent/config.py). Values are read from environment variables with sensible defaults. Constructor arguments override env values.

LLM Settings

Env Var	Field	Default	Description
API_KEY	api_key	""	API key for the OpenAI-compatible endpoint
BASE_URL	base_url	http://localhost:8000/v1	Base URL for chat and embeddings
DEFAULT_MODEL	default_model	qwen3-vl	Default model for chat completions
EMBEDDING_MODEL	embedding_model	text-embedding-3-small	Model name for embeddings

PostgreSQL

Env Var	Field	Default	Description
POSTGRES_HOST	postgres_host	localhost	PostgreSQL host
POSTGRES_PORT	postgres_port	5432	PostgreSQL port
POSTGRES_DB	postgres_db	ldrs_v3	Database name
POSTGRES_USER	postgres_user	ldrs	Database user
POSTGRES_PASSWORD	postgres_password	ldrs_secret	Database password
—	embedding_dim	1536	Embedding vector dimension

LangSmith Monitoring

Env Var	Field	Default	Description
LANGSMITH_TRACING	langsmith_tracing	false	Enable LangSmith tracing
LANGSMITH_API_KEY	langsmith_api_key	""	LangSmith API key
LANGSMITH_PROJECT	langsmith_project	ldrs-v3	LangSmith project name

File Watcher

Env Var	Field	Default	Description
WATCH_DIRS	watch_dirs	./docs	Comma-separated directories to monitor
WATCH_DEBOUNCE	watch_debounce	2.0	Debounce interval in seconds

Retrieval & Agent Tuning

Env Var	Field	Default	Description
MAX_VFS_SECTIONS	max_vfs_sections	15	Max sections in VFS per query
—	max_context_chars	15000	Character budget for context
—	max_grep_results	50	Max TreeGrep results per document
MAX_AGENT_ITERATIONS	max_agent_iterations	10	Max agent loop iterations

Fusion Weights (Defaults)

Field	Default	Description
bm25_weight	0.4	Default BM25 weight
vector_weight	0.4	Default vector similarity weight
grep_weight	0.2	Default grep density weight

These defaults are overridden by the intent-specific presets in FusionRanker.

Directories

Field	Default	Description
results_dir	./results	Structure JSONs and registry
docs_dir	./docs	Source .md files
sessions_dir	./sessions	VFS session data
registry_path	./results/registry.json	Auto-derived from results_dir

Derived Properties

• postgres_dsn — full PostgreSQL connection string
• async_postgres_dsn — async-compatible connection string (for asyncpg)

---

API Reference

The FastAPI server runs on port 8001 (configurable via PORT env var).

POST /query

Run the full 6-stage pipeline for a user query.

Request body:

{
  "query": "How does OAuth2 token refresh work?",
  "conversation_summary": "optional prior conversation summary",
  "recent_turns": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "db_context": null,
  "cleanup_session": false
}

Response:

{
  "answer": "OAuth2 token refresh works by...",
  "intent_type": "conceptual",
  "selected_files": ["authentication.md"],
  "candidates_count": 12,
  "ranked_count": 8,
  "session_id": "a1b2c3d4e5f6",
  "citations": ["authentication.md § Token Refresh"],
  "claims_checked": 3,
  "claims_supported": 3,
  "claims_flagged": 0,
  "re_grounded": false,
  "usage": {
    "total_input_tokens": 2500,
    "total_output_tokens": 800,
    "total_tokens": 3300,
    "total_cost_usd": 0.0,
    "total_llm_calls": 4,
    "total_query_time_ms": 3200.5,
    "stage_breakdown": {},
    "stage_timings_ms": {}
  },
  "total_time_ms": 3450.2,
  "success": true,
  "error": ""
}

POST /batch-query

Run multiple queries sequentially.

Request body:

{
  "queries": ["question 1", "question 2"],
  "conversation_summary": null,
  "cleanup_sessions": false
}

Response: List[QueryResponse]

POST /index

Index a single markdown file.

Request body:

{
  "md_path": "/absolute/path/to/document.md",
  "tags": ["auth", "security"],
  "summary": "OAuth2 authentication documentation",
  "if_thinning": false
}

Response:

{
  "md_path": "/absolute/path/to/document.md",
  "doc_name": "document",
  "index_path": "results/document_structure.json",
  "node_count": 15,
  "section_count": 12,
  "embedded_count": 12,
  "success": true,
  "error": ""
}

POST /index-directory

Index all .md files in a directory.

Request body:

{
  "directory": null,
  "tags": null,
  "if_thinning": false
}

If directory is null, defaults to config.docs_dir.

Response: List[IndexResponse]

GET /corpus

Get a summary of the current corpus.

Response:

{
  "total_files": 5,
  "total_tokens": 25000,
  "total_nodes": 120,
  "files_with_embeddings": 5,
  "file_names": ["auth.md", "api.md", "setup.md"]
}

GET /corpus/stats

Detailed corpus statistics including per-file info.

Response:

{
  "summary": { "...corpus summary..." },
  "files": {
    "auth.md": {
      "summary": "OAuth2 flow documentation",
      "tags": ["auth"],
      "sections_count": 8,
      "size_tokens": 1200,
      "has_embeddings": true,
      "last_modified": "2026-02-01"
    }
  }
}

POST /corpus/rebuild

Re-index all documents in the docs directory. Returns List[IndexResponse].

GET /sessions

List all VFS session IDs. Returns List[str].

DELETE /sessions/{session_id}

Delete a specific VFS session.

Response: {"status": "deleted", "session_id": "..."}

POST /sessions/cleanup

Delete all VFS sessions.

Response: {"status": "cleaned", "sessions_deleted": "5"}

GET /health

Health check endpoint.

Response:

{
  "status": "ok",
  "pipeline_started": true,
  "corpus_files": 5,
  "version": "3.0.0"
}

---

Streamlit UI

The Streamlit chat interface (ui/streamlit_app.py) provides:

• Chat interface — send queries and view cited answers
• Pipeline details — expandable panel showing intent type, candidate counts, ranked counts, claims checked/flagged, timing
• Corpus info — sidebar showing file count, token count, node count
• Usage stats — expandable JSON view of per-stage token usage
• Health status — sidebar indicator of API connection status

Launch:

streamlit run ui/streamlit_app.py

The UI connects to the FastAPI server (default http://localhost:8001). The API URL is configurable via the sidebar text input.

---

Module Reference

All modules live under agent/. The package uses lazy imports via __getattr__ in agent/__init__.py to avoid pulling in heavy dependencies on package load.

config.py — AgentConfig

Centralized @dataclass configuration. All settings loaded from environment variables with sensible defaults. Provides postgres_dsn and async_postgres_dsn computed properties.

monitoring.py — UsageTracker & LangSmith

• setup_monitoring(config) — configures LangSmith tracing env vars
• UsageTracker — per-query accumulator for token counts, latencies, and costs across all stages
• LLMCallRecord — individual LLM call record
• StageTimer — per-stage timing

registry.py — Registry

JSON-based document registry following the AGENT_SYSTEM schema. Tracks:

• Document metadata (summary, tags, sections, token counts)
• Embedding status
• Last modified dates
• Index paths

Key methods:

• add_file() — add/update a file entry with structure tree
• remove_file() — remove a file entry
• mark_embeddings() — update embedding status
• get_corpus_summary() — aggregate corpus statistics
• get_for_llm() — compact version for Stage 1 LLM context
• save() — atomic write (tmp file + rename)

Uses NFC normalization on all text fields and tiktoken for token counting.

embedder.py — Embedder

Async section-level embedding pipeline:

• PostgreSQL connection pool via asyncpg (min 2, max 10 connections)
• Batch embedding via OpenAI-compatible /v1/embeddings endpoint
• Upsert semantics (delete + insert per document)
• Cosine similarity search with optional doc_names scope
• Multi-query search with deduplication

indexer.py — Indexer

End-to-end document indexing:

1. md_to_tree() — parse .md into structure tree via PageIndex
2. Save structure JSON to results/
3. Flatten tree into sections (with breadcrumbs)
4. Embed sections into pgvector
5. Register in Registry with metadata

Also handles remove_file() and index_directory().

tree_grep.py — TreeGrep

Hierarchical pattern search across structure JSON nodes. Three-tier matching with configurable relevance scores:

Field	Relevance Score
Title	3.0
Summary	2.0
Body text	1.0

Two matching modes:

• Tier 1: Exact substring match (full score)
• Tier 2: Word-level match with ratio scaling (min ratio: 0.3)

Features:

• NFC normalization for Unicode safety
• Stop word filtering (200+ English stop words)
• Scope filtering by node_id or title
• Snippet extraction with configurable padding (60 chars)
• search_from_hints() for Intent Classifier pattern_hints
• Multi-pattern search with deduplication

watcher.py — FileWatcher

File system monitoring using watchdog:

• Debounced event handling (configurable interval)
• Handles create, modify, delete, and move events
• Async indexing via asyncio.run_coroutine_threadsafe()
• Automatic registry timestamp updates
• Recursive directory watching

intent_classifier.py — IntentClassifier

Stage 1 LLM call that produces:

• IntentResult with intent type, file selection, query variants
• PatternHints for TreeGrep routing
• SelectedFile list with confidence scores

Handles: malformed JSON, missing fields, invalid intent types. Falls back to conceptual intent with original query on any failure.

retriever.py — Retriever

Stage 2 parallel retrieval:

• Loads TreeGrep instances for selected files
• Runs grep and vector search concurrently via asyncio.gather()
• Merges results into SectionCandidate pool
• Deduplication by (doc_name, section_id)
• Combined signals: grep_score, grep_hits, vector_similarity

fusion_ranker.py — FusionRanker

Stage 3 BM25 fusion ranking:

• BM25Okapi scoring from rank-bm25 library
• BM25 normalization to [0, 1] range
• Intent-based weight presets
• Recency factor (exponential decay, 365-day half-life)
• Tag overlap boost
• Multi-file interleave for comparative intent
• Capped at max_vfs_sections

vfs.py — VFS

Stage 4 virtual filesystem:

• Creates session directories with UUID-based IDs
• Writes ranked sections as individual .md files
• Builds manifest.json with metadata and score breakdowns
• Writes conversation context and db_context
• Supports on-demand section fetching (add_fetched_section)
• Session listing and cleanup

tools.py — AgentTools

Five tools available to the agent during Stage 5:

Tool	Description
read_section(vfs_path)	Read a section from the VFS manifest
fetch_section(source_file, section_header)	Pull additional section on demand
search_conversation_history(query)	Search prior conversation turns
write_scratchpad(content)	Write to private working memory
read_scratchpad()	Read working memory

Tool definitions are generated in OpenAI function-calling format via get_tool_definitions(). The sections_read property tracks which sections were accessed (for citation validation).

agent_loop.py — AgentLoop

Stage 5 iterative function-calling loop:

• System prompt defines the agent protocol (manifest-first, cite-inline)
• Iterates up to max_agent_iterations times
• Each iteration: LLM call with tools → execute tool calls → append results
• Terminates when the LLM produces content without tool calls
• Forced synthesis on iteration limit
• Citation extraction via regex: [source: file.md § Section]
• Records per-iteration token usage

grounding.py — GroundingVerifier

Stage 6 grounding verification:

• Extracts (claim, citation) pairs from answer text
• Locates cited source content in the VFS manifest
• Per-claim LLM entailment check (strict verification prompt)
• Flags unsupported claims (max 10 claims checked per answer)
• Caveat insertion for flagged claims
• Re-grounding trigger at >40% flag ratio
• Hallucination logging to results/hallucination_log.jsonl
• Handles JSON parse failures and verification errors gracefully

pipeline.py — Pipeline

End-to-end orchestrator:

• Lazy component initialization (all components created on first use)
• startup() / shutdown() lifecycle management
• query() runs all 6 stages sequentially
• Re-grounding loop with feedback to the agent
• Conversation state management (last 20 turns)
• Index pass-through (index_file, index_directory)
• Corpus info and session management utilities

---

Indexing & File Watcher

Document Format

Source documents are Markdown files. PageIndex parses them into a hierarchical structure tree based on heading levels.

Indexing Pipeline

.md file → md_to_tree() → structure JSON → flatten → embed → register

1. PageIndex parsing — md_to_tree() converts Markdown headings into a tree structure with node IDs, titles, text content, and page ranges.

2. Structure JSON — saved to results/<doc_name>_structure.json for TreeGrep and the fetch_section tool.

3. Flattening — the tree is flattened into a list of section dicts with {node_id, title, text, line_num, breadcrumb}. Only nodes with non-empty text are included.

4. Embedding — sections are batch-embedded via the OpenAI-compatible embeddings endpoint and upserted into the sections table in PostgreSQL with pgvector.

5. Registration — the document is added to registry.json with metadata (summary, tags, sections, token count, embedding status).

File Watcher

The watcher monitors directories for .md file changes:

from agent.watcher import FileWatcher
from agent.config import AgentConfig

config = AgentConfig()
watcher = FileWatcher(config)
await watcher.start()   # starts watchdog observer
# ... application runs ...
await watcher.stop()     # stops and cleans up

Events handled:

• created / modified → re-index the file
• deleted → remove embeddings + registry entry + structure JSON
• moved → treat source as deleted, destination as created

---

Virtual Filesystem (VFS)

Each query creates a VFS session directory:

sessions/{session_id}/
  manifest.json              ← agent reads this first
  retrieved/
    rank1__docname__section.md
    rank2__docname__section.md
    ...
  conversation/
    history_summary.md        ← conversation context
    recent_turns.json         ← last N turns
  db_context/
    relevant_records.json     ← optional database results
  working/
    scratchpad.md             ← agent working memory

manifest.json

{
  "session_id": "a1b2c3d4e5f6",
  "created_at": "2026-02-28T10:00:00Z",
  "intent_type": "conceptual",
  "query_variants": ["original query", "variant 1"],
  "sections": [
    {
      "vfs_path": "retrieved/rank1__auth__oauth_flow.md",
      "source_file": "authentication.md",
      "section": "OAuth Flow",
      "one_line_summary": "Describes the OAuth2 authorization code flow...",
      "retrieval_method": "grep+vector",
      "final_score": 0.85,
      "score_breakdown": {
        "bm25": 0.7,
        "vector": 0.9,
        "grep_density": 0.3
      },
      "why_included": "vector=0.90; bm25=0.70; grep_density=0.30",
      "last_modified": "2026-02-01",
      "fetch_more_hint": false
    }
  ]
}

---

Agent Tools & Function Calling

The agent has five tools available, defined in OpenAI function-calling format:

read_section

{
  "name": "read_section",
  "parameters": {
    "vfs_path": "retrieved/rank1__auth__oauth_flow.md"
  }
}

Reads a section listed in the manifest. Only sections read via this tool may be cited. The tool tracks all accessed paths in sections_read.

fetch_section

{
  "name": "fetch_section",
  "parameters": {
    "source_file": "authentication.md",
    "section_header": "Token Refresh"
  }
}

Pulls an additional section on demand from the structure JSON. The section is added to the VFS retrieved/ directory and manifest. Use when the manifest hints at more content or a multihop reference is followed.

search_conversation_history

{
  "name": "search_conversation_history",
  "parameters": {
    "query": "what did we discuss about authentication"
  }
}

Searches prior conversation turns via keyword matching.

write_scratchpad

{
  "name": "write_scratchpad",
  "parameters": {
    "content": "## Reasoning\n..."
  }
}

Writes to the agent's private working memory. Content should follow the structured format: ## Reasoning, ## Key Facts, ## Open Questions, ## Synthesis Plan.

read_scratchpad

No parameters. Reads the current scratchpad content.

---

Grounding Verification

Process

1. Claim extraction — regex matches sentences followed by [source: ...] citations
2. Source lookup — maps citations to VFS sections via manifest
3. Entailment check — LLM verifies each claim against its source
4. Caveat insertion — unsupported claims get a [Note: This claim could not be fully verified...] suffix
5. Re-grounding — if >40% of claims are flagged, the pipeline re-runs Stage 5 with explicit feedback about which claims failed

Hallucination Log

Flagged claims are appended to results/hallucination_log.jsonl:

{
  "timestamp": "2026-02-28T10:00:00Z",
  "session_id": "a1b2c3d4e5f6",
  "claim": "The refresh token expires after 30 days",
  "citation": "authentication.md § Token Refresh",
  "supported": false,
  "reason": "The source does not mention a 30-day expiry period"
}

Cost Control

• Maximum 10 claims verified per answer (MAX_CLAIMS_TO_VERIFY)
• Source content truncated to 3000 chars per verification call
• Re-grounding runs at most once (MAX_REGROUND_ATTEMPTS = 1)

---

Monitoring & LangSmith

UsageTracker

Every query creates a UsageTracker that records:

• Per-call: stage, model, input/output tokens, latency, cost
• Per-stage: start/end timing
• Query-level: total query time

The summary() method returns a dict with:

{
    "total_input_tokens": 2500,
    "total_output_tokens": 800,
    "total_tokens": 3300,
    "total_cost_usd": 0.0,
    "total_llm_calls": 4,
    "total_query_time_ms": 3200.5,
    "stage_breakdown": {
        "intent_classifier": {"input_tokens": 500, "output_tokens": 200, ...},
        "agent_loop": {"input_tokens": 1500, "output_tokens": 400, ...},
        "grounding": {"input_tokens": 500, "output_tokens": 200, ...},
    },
    "stage_timings_ms": {
        "intent_classifier": 850.2,
        "retrieval": 200.5,
        "fusion_ranking": 15.3,
        "vfs_population": 25.1,
        "agent_loop": 1800.0,
        "grounding": 310.5,
    },
}

LangSmith Integration

Enable LangSmith tracing via environment variables:

LANGSMITH_TRACING=true
LANGSMITH_API_KEY=your-key
LANGSMITH_PROJECT=ldrs-v3

setup_monitoring(config) sets the appropriate environment variables that LangSmith/LangChain read automatically. Must be called before any LangChain model is instantiated (handled by Pipeline.startup()).

---

Citation Format

The agent uses inline citations in its answers:

Single source:

OAuth2 uses refresh tokens to obtain new access tokens. [source: auth.md § Token Refresh]

Multiple sources:

Both OAuth2 and API keys support scoped permissions. [source: auth.md § Scopes, api_keys.md § Permissions]

Citations are extracted by the AgentLoop._extract_citations() method using the regex pattern \[source:\s*([^\]]+)\].

The grounding verifier parses citations in the format file.md § Section Name to locate source content in the VFS.

---

Testing

Running Tests

# From the project root
python -m pytest tests/ -v

# Or with the development install
pytest tests/ -v

Test Suite

• 74 tests covering all modules and pipeline stages
• All tests pass in approximately 1.5 seconds
• Fully offline — no LLM or database calls
• Uses unittest.mock.AsyncMock for async mocking
• Uses pytest-asyncio with asyncio_mode = "auto"
• Test fixtures use tmp_path for filesystem isolation

Test Coverage

Tests cover:

• AgentConfig — default values, environment overrides, DSN properties
• UsageTracker — recording, timing, summary computation
• Registry — add/remove files, save/load, corpus summary, LLM format
• TreeGrep — exact/word-level matching, regex, scope, multi-pattern, search_from_hints, deduplication
• IntentClassifier — response parsing, malformed JSON, empty registry
• SectionCandidate — signal merging
• FusionRanker — weight presets, multi-file interleave, recency/tag boosts
• VFS — session creation, manifest, section read/write, scratchpad, add fetched section, cleanup
• AgentTools — all five tools, error handling
• AgentLoop — citation extraction
• GroundingVerifier — claim extraction, verification parsing, caveating, flag logging
• Pipeline — lifecycle, conversation management, corpus info
• Indexer — section flattening, breadcrumbs, recursive node counting

Test Data

• tests/fixtures/ — 9 structure JSON files for TreeGrep testing
• tests/markdown/ — 12 .md files for indexer testing

---

Project Structure

ldrs_v3/
├── README.md                    ← this file
├── pyproject.toml               ← project metadata, dependencies, tool config
├── requirements.txt             ← pinned dependencies
├── .env.example                 ← environment variable template
├── .gitignore
├── docker-compose.yml           ← PostgreSQL + pgvector (pg16)
│
├── agent/                       ← core pipeline modules
│   ├── __init__.py              ← lazy imports, __all__ exports
│   ├── config.py                ← AgentConfig dataclass
│   ├── monitoring.py            ← UsageTracker, LangSmith setup
│   ├── registry.py              ← document registry (JSON)
│   ├── embedder.py              ← pgvector embedding pipeline
│   ├── indexer.py               ← md → tree → embed → register
│   ├── tree_grep.py             ← hierarchical pattern search
│   ├── watcher.py               ← file system watcher (watchdog)
│   ├── intent_classifier.py     ← Stage 1: intent + routing
│   ├── retriever.py             ← Stage 2: parallel retrieval
│   ├── fusion_ranker.py         ← Stage 3: BM25 fusion ranking
│   ├── vfs.py                   ← Stage 4: VFS population
│   ├── tools.py                 ← Stage 5: agent tools
│   ├── agent_loop.py            ← Stage 5: agent reasoning loop
│   ├── grounding.py             ← Stage 6: grounding verification
│   └── pipeline.py              ← end-to-end orchestrator
│
├── api/
│   ├── __init__.py
│   └── server.py                ← FastAPI server (11 endpoints)
│
├── ui/
│   ├── __init__.py
│   └── streamlit_app.py         ← Streamlit chat interface
│
├── pageindex/                   ← PageIndex library (md_to_tree)
│   ├── __init__.py
│   ├── page_index_md.py         ← markdown → structure tree
│   ├── utils.py                 ← shared utilities
│   └── config.yaml              ← PageIndex configuration
│
├── scripts/
│   └── init_db.sql              ← PostgreSQL schema (sections + hallucination_log)
│
├── tests/
│   ├── test_ldrs_v3.py          ← 74 tests (all passing)
│   ├── fixtures/                ← 9 structure JSON test files
│   └── markdown/                ← 12 .md test files
│
├── docs/                        ← source .md files (user content)
├── results/                     ← runtime: structure JSONs + registry.json
└── sessions/                    ← runtime: VFS session directories

---

Database Schema

PostgreSQL 16 with pgvector extension. Tables created by scripts/init_db.sql:

sections table

Column	Type	Description
id	SERIAL PRIMARY KEY	Auto-incrementing ID
doc_name	TEXT NOT NULL	Document name
section_id	TEXT NOT NULL	node_id from PageIndex
section_title	TEXT NOT NULL	Section title
source_file	TEXT NOT NULL	Path to source .md file
content	TEXT NOT NULL	Full section text
line_num	INTEGER	Line number in source .md
embedding	vector(1536)	Embedding vector
token_count	INTEGER DEFAULT 0	Token count
created_at	TIMESTAMPTZ	Creation timestamp
updated_at	TIMESTAMPTZ	Update timestamp

Unique constraint: (doc_name, section_id)

Indexes:

• idx_sections_embedding — IVFFlat cosine index (100 lists)
• idx_sections_doc_name — B-tree on doc_name

hallucination_log table

Column	Type	Description
id	SERIAL PRIMARY KEY	Auto-incrementing ID
session_id	TEXT NOT NULL	VFS session ID
claim_text	TEXT NOT NULL	The flagged claim
cited_source	TEXT	Citation reference
cited_section	TEXT	Section reference
supported	BOOLEAN DEFAULT FALSE	Verification result
confidence	FLOAT	Confidence score
logged_at	TIMESTAMPTZ	Timestamp

---

Troubleshooting

PostgreSQL connection errors

Ensure Docker is running and the pgvector container is healthy:

docker compose ps
docker compose logs postgres

Verify the database is accessible:

psql postgresql://ldrs:ldrs_secret@localhost:5432/ldrs_v3

Embedding dimension mismatch

The default embedding dimension is 1536 (matching text-embedding-3-small). If using a different model, update both:

1. embedding_dim in AgentConfig
2. The vector(1536) column type in init_db.sql

Empty search results

• Verify documents are indexed: GET /corpus
• Check that structure JSONs exist in results/
• Ensure embeddings were generated: files_with_embeddings > 0
• Check logs for embedding errors

Agent loop hitting max iterations

Increase MAX_AGENT_ITERATIONS or check if the manifest has enough sections. The agent may be trying to fetch sections that don't exist.

LangSmith tracing not working

Ensure all three variables are set:

LANGSMITH_TRACING=true
LANGSMITH_API_KEY=<valid-key>
LANGSMITH_PROJECT=ldrs-v3

And that setup_monitoring() is called before any LLM calls (handled automatically by Pipeline.startup()).

Unicode / Nepali text issues

All text processing applies unicodedata.normalize("NFC", text) at boundaries. If you see matching failures with Devanagari text, check that:

1. Source files are saved as UTF-8
2. NFC normalization is applied before comparison
3. The database encoding is UTF-8

---

License

MIT