Incremental semantic index ingestion.

[Mpdreamz]

This PR documents our Elasticsearch integration architecture, which implements an incremental indexing strategy for maintaining both lexical (traditional full-text) and semantic (vector embeddings) search indices.

The Problem We're Solving

Every time we rebuild our documentation, we need to:

• Index potentially thousands of markdown documents
• Generate expensive semantic embeddings for semantic search
• Keep two separate indices (lexical + semantic) in sync
• Avoid reprocessing unchanged documents

The Solution: Hash-Based Change Detection + Incremental Sync

During Indexing (Lexical Index):

• We compute a hash from each document's URL, body, and headings
• If the hash matches what's already indexed → only update batch_index_date
• If the hash changed → full document upsert with new last_updated timestamp
• This means we track every document in the current batch, but only fully update what actually changed.

The hash checking happens on the server (elasticsearch) through the reindex API.

Synchronize from lexical to semantic index

The StopAsync method orchestrates a 5-phase synchronization:

1. Finalize lexical index - Drain buffers, refresh, apply aliases
2. Bootstrap semantic index - Create if it doesn't exist yet
3. Sync updates - Copy only documents where last_updated >= batch_date from lexical → semantic

• Only changed docs get re-embedded

4. Sync deletions - Reindex from lexical to semantic where: batch_index_date < batch_date and make those operations a delete.
5. Cleanup - Delete stale docs batch_index_date < batch_date from lexical index itself.

Why This Matters

• Massive performance gains: If only 10 out of 10,000 docs changed, we only regenerate 10 semantic embeddings
• Cost savings: Semantic embedding generation is expensive (compute + potential API costs)
• Zero-downtime: Time-stamped indices with alias swapping means no search interruption
• Data consistency: Both indices stay perfectly synchronized despite being updated separately
• Change tracking, we can use last_updated to boost recency (if needed).

Visualization

New docs includes a step-by-step visual walkthrough showing how documents flow through the system, with color-coded states (blue = batch-tracked, green = updated, red = marked for deletion).

For Reviewers

• ElasticsearchExporter.cs - Base exporter with channel management
• ElasticsearchMarkdownExporter.cs - Orchestration logic and sync phases

[github-actions]

🔍 Preview links for changed docs

• docs/development/ingest/index.md

[reakaleek]

Nice visualizations!