Merge dev into main for v1.1.0 release

Author: castnettech

.gitignore

@@ -54,6 +54,9 @@ docs/
 !docs/benchmark_meat_detection_query.md
 !docs/assets/
 !docs/assets/mnemosyne.png
+!docs/assets/diagrams/
+!docs/assets/diagrams/*.png
+!docs/assets/diagrams/*.gif
 
 # -- Holdback test data (never committed -- validation-only) --
 mnemosyne/tests/benchmark_questions/httpx_holdback.json

CHANGELOG.md

@@ -5,6 +5,90 @@ All notable changes to Mnemosyne are documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.0] - 2026-04-07
+
+### Added
+- Document ingestion (Tier 0) -- extract and index PDFs, DOCX, CSV, and
+  plaintext (.log, .cfg, .ini, .conf, .rst, .xml). PDF requires optional
+  `mnemosyne-engine[pdf]` extra (pypdf, pure Python, BSD). All other
+  extractors use stdlib only. Zero-dep core preserved.
+- Partitioned document index -- documents stored in isolated tables
+  (doc_chunks, doc_chunks_fts, doc_sparse_embeddings, doc_vocabulary) within
+  the same SQLite database. Code retrieval pipeline is completely untouched.
+  New DocStore and DocRetrievalEngine with BM25 + TF-IDF fused via RRF.
+- Schema ingestion pipeline -- DDL-aware SQL chunking (Phase 1), JSON/YAML
+  schema import with environment tagging (Phase 2), SQLite introspection via
+  PRAGMA queries with `--sqlite` CLI flag (Phase 3).
+- CLI `--docs` flag -- search document partition only.
+- CLI `--all` flag -- search both code and document partitions with split budget.
+- CLI `schema-ingest` and `schema-stats` commands for database DDL ingestion.
+- CLI `--version` flag -- reads from package `__version__`.
+- One-time upgrade hint -- on first query or ingest after schema migration,
+  a stderr message reminds users about the new `--all` and `--docs` flags.
+  Fires once, then clears. Fresh installs are not affected.
+- Progress bar on `mnemosyne ingest` with live counter and file path display.
+- Document partition benchmark -- 18-test regression gate covering document
+  retrieval, schema lookup, cross-partition queries, and partition isolation.
+
+### Fixed
+- Document file size limit -- documents were silently skipped using the 512 KB
+  code limit. Now uses extraction.max_file_size_kb (default 10 MB).
+- DocStore vocabulary adapter -- doc_tfidf was created with store=None at all
+  call sites, so doc_vocabulary was never populated. Added save_vocabulary()
+  and load_vocabulary() adapter methods, wired all call sites. Re-ingest
+  required after upgrade to populate document vocabulary.
+
+### Changed
+- .md and .txt files reclassified as documents -- route to the doc partition
+  instead of competing with source code for ranking slots. Use `--all` to
+  search both partitions.
+- Dotfile ignore consolidation -- replaced 12 individual patterns with a
+  single `.*` glob that catches all dotfiles and dotdirs.
+- Schema migration 2->3: files table gains extraction_method,
+  extraction_quality, page_count; chunks table gains page_number.
+- Schema migration 3->4: creates doc_chunks, doc_chunks_fts,
+  doc_sparse_embeddings, doc_vocabulary tables.
+
+### Upgrade notes
+- Run `mnemosyne ingest --full` after upgrading to populate document
+  partitions and vocabulary. Existing code indexes are preserved.
+- Queries that previously found .md files via default `mnemosyne query` must
+  now use `--all` or `--docs`.
+
+---
+
+### mnemosyne-mcp [0.2.0]
+
+### Added
+- `search_docs` tool -- document-only MCP search for LLM agents.
+- `schema_ingest` tool -- import DDL files or introspect SQLite databases.
+- `schema_stats` tool -- report indexed schema sources and statistics.
+- Federated search -- `search` tool now queries both code and document
+  partitions, returns labeled sections ("Code Results" / "Document Results").
+
+### Fixed
+- Per-partition truncation -- the 64KB output cap was applied after
+  concatenation, silently removing document results when code results
+  exceeded the cap. Each partition now gets an independent budget (32KB each
+  when both exist, full 64KB when only one exists).
+- Stats handler tuple unpack -- _handle_stats unpacked 3 values but
+  _get_engine returns 5 since the partition refactor.
+- doc_tfidf store wiring at 2 server.py call sites.
+
+---
+
+### mnemosyne-ollama [0.1.1]
+
+### Added
+- Partition visibility in verbose mode -- `-v` output shows code/docs chunk
+  counts and top file paths per partition for `search` and `search_docs` calls.
+
+### Fixed
+- Version flag now imports from package `__version__` instead of hardcoded
+  string.
+
+---
+
 ## [1.0.5] - 2026-04-04
 
 ### Fixed
@@ -137,6 +221,8 @@ Initial public release on PyPI as `mnemosyne-engine`.
 - Daemon mode with Unix socket RPC
 - Zero runtime dependencies
 
+[1.1.0]: https://github.com/castnettech/mnemosyne/compare/v1.0.5...v1.1.0
+[1.0.5]: https://github.com/castnettech/mnemosyne/compare/v1.0.4...v1.0.5
 [1.0.4]: https://github.com/castnettech/mnemosyne/compare/v1.0.2...v1.0.4
 [1.0.2]: https://github.com/castnettech/mnemosyne/compare/v1.0.0...v1.0.2
 [1.0.0]: https://github.com/castnettech/mnemosyne/compare/v0.4.0...v1.0.0

README.md

@@ -17,7 +17,11 @@
 
 ---
 
-Mnemosyne indexes your codebase into a local SQLite store, scores every chunk with a 6-signal hybrid retriever, compresses results with AST awareness, and returns exactly what you need within a token or result budget. It runs entirely locally -- no API keys, no cloud, no runtime dependencies beyond Python 3.11+.
+<p align="center">
+  <img src="https://raw.githubusercontent.com/castnettech/mnemosyne/main/docs/assets/diagrams/mnemosyne-ecosystem.gif" alt="Mnemosyne Ecosystem -- three packages, zero cloud" width="800">
+</p>
+
+Mnemosyne indexes your codebase and documents into a local SQLite store, scores every chunk with a 6-signal hybrid retriever, compresses results with AST awareness, and returns exactly what you need within a token or result budget. Supports source code (Python, JS/TS, Go, Rust, C#, Java, Kotlin), documents (PDF, DOCX, CSV, plaintext), and database schemas (SQL DDL, JSON snapshots, SQLite introspection). It runs entirely locally -- no API keys, no cloud, no runtime dependencies beyond Python 3.11+.
 
 ## Install
 
@@ -53,7 +57,9 @@ mnemosyne query "How does authentication work?"   # search
 - **Self-tuning ARC cache** -- adapts between recency and frequency patterns automatically, persisted across sessions
 - **Delta-aware tracking** -- detects file and chunk-level changes, delivers diffs instead of full content (80-95% savings on incremental queries)
 - **Content deduplication** -- SHA-256 addressed storage eliminates duplicate chunks across files
-- **7-language structural chunking** -- Python (AST), JavaScript/TypeScript, Go, C#, Rust, Java, Kotlin, plus Markdown and plain text
+- **7-language structural chunking** -- Python (AST), JavaScript/TypeScript, Go, C#, Rust, Java, Kotlin
+- **Document ingestion** -- PDF, DOCX, CSV, and plaintext extraction into an isolated document partition with independent BM25 + TF-IDF retrieval. Optional `mnemosyne-engine[pdf]` extra for PDF support
+- **Schema ingestion** -- DDL files, JSON/YAML snapshots, and live SQLite introspection indexed alongside code for cross-domain queries
 - **Daemon mode** -- JSON-RPC over Unix socket keeps indexes warm for sub-20ms queries
 - **Full audit trail** -- append-only JSON-lines log of every operation
 - **Zero runtime dependencies** -- pure Python 3.11+ stdlib. One `pip install`, no conflicts
@@ -121,8 +127,10 @@ Works with any agent that can execute shell commands.
 |---|---|
 | `init` | Create workspace and config |
 | `ingest` | Index files (incremental, `--full` to rebuild) |
-| `query` | Search with token budget |
+| `query` | Search with token budget (`--docs`, `--all` for document partition) |
 | `stats` | Index and cache statistics |
+| `schema-ingest` | Import database schema (DDL, JSON, SQLite) |
+| `schema-stats` | Schema index statistics |
 | `compress` | Preview compression for a file |
 | `delta` | Show changes since last index |
 | `cache` | Manage ARC cache (`show`, `clear`, `warm`) |

docs/assets/diagrams/mnemosyne-ecosystem.gif

@@ -1,6 +1,6 @@
 # mnemosyne-mcp
 
-MCP server for [Mnemosyne](https://github.com/castnettech/mnemosyne-engine) -- a 6-signal hybrid code retrieval engine that reduces LLM context waste by 73%.
+MCP server for [Mnemosyne](https://github.com/castnettech/mnemosyne) -- a 6-signal hybrid retrieval engine for code, documents, and database schemas. Reduces LLM context waste by 73%.
 
 For the full reference, see [MCP.md](../MCP.md) in the repository root.
 
@@ -31,33 +31,58 @@ Or add to your project's `.mcp.json`:
 
 ## Tools
 
-### `mnemosyne.search`
+### `search`
 
-Search a codebase using 6-signal hybrid retrieval (BM25 + TF-IDF + symbol matching + usage frequency + predictive prefetch + optional dense embeddings) fused via Reciprocal Rank Fusion. Returns the most relevant code chunks within a configurable token budget, with AST-aware compression.
+Federated search across code and document partitions. Code results use 6-signal hybrid retrieval (BM25, TF-IDF, symbol matching, usage frequency, predictive prefetch, optional dense embeddings) fused via Reciprocal Rank Fusion. Document results use BM25 + TF-IDF with isolated vocabulary. Returns labeled sections so the LLM can perform cross-type ranking.
 
 **Parameters:**
 - `query` (string, required) -- natural language or keyword query
 - `budget` (integer, default 8000) -- maximum token budget
 - `project_root` (string, optional) -- path to project root
 
-### `mnemosyne.index`
+### `search_docs`
 
-Index or re-index a codebase. Incremental by default (only processes changed files).
+Search the document partition only (PDFs, DOCX, CSVs, logs, and other non-code files). Uses BM25 and TF-IDF with an isolated vocabulary tuned for prose retrieval.
+
+**Parameters:**
+- `query` (string, required) -- natural language query
+- `budget` (integer, default 8000) -- maximum token budget
+- `project_root` (string, optional) -- path to project root
+
+### `index`
+
+Index or re-index a codebase. Incremental by default (only processes changed files). Indexes both code and document partitions.
 
 **Parameters:**
 - `project_root` (string, optional) -- path to project root
 - `full` (boolean, default false) -- force full re-index
 
-### `mnemosyne.stats`
+### `stats`
 
 Show index statistics: file count, chunk count, tokens, language breakdown, chunk types.
 
 **Parameters:**
 - `project_root` (string, optional) -- path to project root
 
+### `schema_ingest`
+
+Ingest database schema into the index. Accepts DDL files (.sql), JSON/YAML schema snapshots, or SQLite database paths for live introspection.
+
+**Parameters:**
+- `source_path` (string, required) -- path to schema source
+- `environment` (string, optional) -- tag for the schema source (e.g., "production", "staging")
+- `project_root` (string, optional) -- path to project root
+
+### `schema_stats`
+
+Report indexed schema sources and statistics.
+
+**Parameters:**
+- `project_root` (string, optional) -- path to project root
+
 ## How it works
 
-Mnemosyne indexes your codebase into local SQLite, scoring every chunk with six retrieval signals fused through Reciprocal Rank Fusion. AST-aware compression then strips boilerplate while preserving function signatures, control flow, and documentation -- delivering exactly within your token budget.
+Mnemosyne indexes your codebase and documents into local SQLite, scoring every chunk with retrieval signals fused through Reciprocal Rank Fusion. Code gets 6-signal hybrid search; documents get BM25 + TF-IDF with isolated vocabulary. AST-aware compression strips boilerplate while preserving function signatures, control flow, and documentation.
 
 Zero runtime dependencies beyond Python 3.11+. No API keys. No cloud services. Everything runs locally.
 

mcp/README.md

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mnemosyne-mcp"
-version = "0.1.0"
+version = "0.2.0"
 description = "MCP server for Mnemosyne -- semantic code retrieval with 6-signal hybrid search and AST-aware compression"
 readme = "README.md"
 license = "AGPL-3.0-only"

mcp/pyproject.toml

@@ -1,3 +1,3 @@
 """Mnemosyne MCP Server -- semantic code retrieval for Claude Code."""
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"