overhaul, mvp

Signed-off-by: allen-munsch <james.a.munsch@gmail.com>

Author: allen-munsch

Makefile

@@ -130,6 +130,32 @@ shell:
 iex:
 	@docker compose exec coordinator bin/semantic_fabric remote
 
+# Local Development Commands (using asdf)
+.PHONY: local-deps local-compile local-test local-run local-format local-iex
+local-deps:
+	@echo "Fetching local dependencies..."
+	@mix deps.get
+
+local-compile:
+	@echo "Compiling local project..."
+	@mix compile
+
+local-test:
+	@echo "Running local test suite..."
+	@mix test
+
+local-run:
+	@echo "Running local application..."
+	@mix run --no-halt
+
+local-format:
+	@echo "Formatting local code..."
+	@mix format
+
+local-iex:
+	@echo "Starting local IEx console..."
+	@iex -S mix
+
 # Utility commands
 .PHONY: update-deps
 update-deps:

README.md

@@ -1,79 +1,191 @@
 # MosaicDB
 
-Distributed semantic search built on SQLite shards ( a sketch )
+### A Distributed, Federated Semantic Search Engine Built on SQLite Shards
 
-## Setup
+MosaicDB is an experimental distributed query engine performing **hybrid vector + metadata search** across many **immutable SQLite shard files**. Each shard contains:
 
-```bash
-make build
-make up
-```
+* Document text or metadata
+* Vector embeddings (`sqlite-vss`)
+* PageRank or other ranking signals
+
+Elixir acts as the **coordinator and control plane**, orchestrating fan-out queries, retries, merges, caching, and ranking.
+
+---
+
+# Features
+
+* Federated search across multiple SQLite shards
+* Vector similarity search using `sqlite-vss`
+* Metadata-aware filtering
+* PageRank-based reranking
+* LRU embedding cache
+* Distributed coordinator architecture
+* HTTP API for search
+* Metrics via Prometheus/Grafana
+
+MosaicDB combines **SQLite simplicity with Erlang/Elixir scale**. Each node is a lightweight SQLite database capable of storing both vector embeddings and structured metadata. Distributed across multiple nodes, MosaicDB provides fault-tolerant, scalable storage **without the overhead of managed clusters**.
+
+---
+
+# Feature Comparison
+
+| Feature         | PostgreSQL                    | Pinecone      | Weaviate      | MosaicDB (SQLite nodes)           |
+| --------------- | ----------------------------- | ------------- | ------------- | --------------------------------- |
+| SQL support     | Yes                           | No            | No            | Yes, native SQLite queries        |
+| Vector search   | Extensions needed (pgvector)  | Yes           | Yes           | Yes, exact or approximate         |
+| Distribution    | Manual (sharding/replication) | Managed       | Managed       | Built-in via Elixir/Erlang        |
+| Fault tolerance | Manual / HA setups            | Cloud-managed | Cloud-managed | Erlang/Elixir supervision trees   |
+| Lightweight     | Moderate                      | No            | No            | Each node is a single SQLite file |
+| Edge-ready      | No                            | No            | No            | Yes, nodes are self-contained     |
+
+**Developer Pitch:**
+MosaicDB gives developers a **lightweight, distributed vector + relational database** where each node is just a SQLite file. Fully SQL-capable, fault-tolerant via Erlang/Elixir, and easy to deploy at the edge — you get vector search + relational queries in one place, without complex cluster management or cloud lock-in. It’s **SQLite simplicity with Erlang reliability**.
+
+---
+
+# Why Elixir?
 
-Done. Go to http://localhost/health
+MosaicDB uses Elixir for its coordination layer because it naturally fits **federated query execution**:
 
-## What It Does
+### Concurrency for fan-out search
 
-Takes queries, searches across SQLite database shards, returns ranked results.
+Each shard query runs as an isolated BEAM process—no thread pools, no shared state, no locks.
 
-Each shard = immutable SQLite file with documents, vectors, and PageRank scores.
+### Supervisor-based fault tolerance
 
-This is a system for performing semantic search across many distributed chunks of data (shards) using embedding-based vector search (via sqlite-vss) and then ranking the results.
+Shard errors, timeouts, or node failures are isolated and automatically recovered.
 
-All in a federated/distributed way, inspired by scalable distributed systems like Riak.
+### Predictable under load
 
-## Services
+The BEAM scheduler ensures slow shards do not block others.
 
-- **Coordinator** (4040): Routes queries
-- **Nginx** (80): Load balancer
-- **Redis** (6379): Cache
-- **Prometheus** (9090): Metrics
-- **Grafana** (3000): Dashboards
+### Built-in distribution
+
+Elixir nodes auto-discover and form a cluster, enabling multi-node coordination without external registries.
+
+### Clean pipeline composition
+
+Query planning, merging, and reranking are expressed using functional pipelines and pattern matching.
+
+### Observability
+
+LiveDashboard, telemetry, and introspection tools simplify distributed debugging.
+
+**In short:** Elixir is the resilient, concurrent **control plane** around fast SQLite shards.
+
+---
+
+# Quick Start
+
+## Build and run
+
+```bash
+make build
+make up
+```
 
-## API
+Check health:
 
 ```bash
-# Health
 curl http://localhost/health
+```
 
-# Search (placeholder)
-curl -X POST http://localhost/api/search \
-  -d '{"query":"test"}' \
-  -H "Content-Type: application/json"
+---
+
+# API
+
+### Health
+
+```bash
+curl http://localhost/health
 ```
 
-## Commands
+### Search (placeholder API)
 
 ```bash
-make up       # Start
-make down     # Stop  
-make logs     # Logs
-make restart  # Restart
+curl -X POST http://localhost/api/search \
+  -H "Content-Type: application/json" \
+  -d '{"query": "test"}'
 ```
 
-## Local Dev
+---
+
+# Components
+
+| Service     | Port | Description                |
+| ----------- | ---- | -------------------------- |
+| Coordinator | 4040 | Elixir-based query router  |
+| Nginx       | 80   | Load balancer / entrypoint |
+| Redis       | 6379 | Metadata + embedding cache |
+| Prometheus  | 9090 | Metrics                    |
+| Grafana     | 3000 | Dashboards                 |
+
+---
+
+# Development
+
+Install dependencies:
 
 ```bash
 mix deps.get
+```
+
+Run the system:
+
+```bash
 mix run --no-halt
 ```
 
-## Architecture
+---
+
+# Basic Architecture
+
+```
+Client Query
+      │
+    Nginx
+      │
+  Coordinator (Elixir)
+ ┌────┴─────────────┐
+ │ fan-out async RPC│
+ └────┬─────────────┘
+  Many SQLite Shards
+      │
+  Vector + metadata search
+      │
+  Coordinator merges + ranks
+      │
+    Response
+```
+
+---
+
+# Scaling
 
+To scale horizontally, edit `docker-compose.yml` and increase coordinator workers:
+
+```yaml
+scale: 4
 ```
-Query → Nginx → Coordinator → Shards (SQLite files)
-                    ↓
-                  Cache (Redis)
+
+Then:
+
+```bash
+make restart
 ```
 
-## Scaling
+Elixir nodes will auto-discover each other (via libcluster) and share load.
+
+---
 
-Edit `docker-compose.yml`, add more workers, restart.
+# Documentation
 
-## Docs
+* `docs/ARCHITECTURE.md` — data flow, shard layout, search pipeline
+* `docs/DEPLOYMENT_GUIDE.md` — running MosaicDB in production
+* `docs/SHARD_FORMAT.md` — SQLite schema, embeddings, PageRank structure
 
-- `docs/ARCHITECTURE.md` - How it works
-- `docs/DEPLOYMENT_GUIDE.md` - Production setup
+---
 
-## License
+# License
 
 MIT

config/config.exs

@@ -1,3 +1,7 @@
 import Config
+
+config :nx, :default_backend, EXLA.Backend
+config :bumblebee, :default_backend, EXLA.Backend
+
 config :logger, level: :info
-import_config "#{config_env()}.exs"
+import_config "#{config_env()}.exs"