Merge pull request #18 from christopherkarani/2614

Investigate WAL compaction gains

Author: christopherkarani

README.md

@@ -20,7 +20,7 @@
 <p align="center">
   <img src="https://img.shields.io/badge/Swift-6.2-orange.svg" alt="Swift 6.2">
   <img src="https://img.shields.io/badge/platforms-iOS%2026%20%7C%20macOS%2026-blue.svg" alt="Platforms">
-  <img src="https://img.shields.io/badge/license-MIT-green.svg" alt="License">
+  <img src="https://img.shields.io/badge/license-Apache_2.0-green.svg" alt="License">
 </p>
 
 ---
@@ -104,16 +104,120 @@ Cold Open → First Query: 17ms
 Hybrid Search @ 10K docs: 105ms
 ```
 
+### Core Benchmark Baselines (as of February 17, 2026)
+
+These are reproducible XCTest benchmark baselines captured from the current Wax benchmark harness.
+
+#### Ingest throughput (`testIngestHybridBatchedPerformance`)
+
+| Workload | Time | Throughput |
+|:---|---:|---:|
+| smoke (200 docs) | `0.103s` | `~1941.7 docs/s` |
+| standard (1000 docs) | `0.309s` | `~3236.2 docs/s` |
+| stress (5000 docs) | `2.864s` | `~1745.8 docs/s` |
+| 10k | `7.756s` | `~1289.3 docs/s` |
+
+#### Search latency
+
+| Workload | Time | Throughput |
+|:---|---:|---:|
+| warm CPU smoke | `0.0015s` | `~666.7 ops/s` |
+| warm CPU standard | `0.0033s` | `~303.0 ops/s` |
+| warm CPU stress | `0.0072s` | `~138.9 ops/s` |
+| 10k CPU hybrid iteration | `0.103s` | `~9.7 ops/s` |
+
+#### Recall latency (`testMemoryOrchestratorRecallPerformance`)
+
+| Workload | Time |
+|:---|---:|
+| smoke | `0.103s` |
+| standard | `0.101s` |
+
+Stress recall is currently harness-blocked (`signal 11`) and treated as a known benchmark issue.
+
+#### FastRAG builder
+
+| Mode | Time |
+|:---|---:|
+| fast mode | `0.102s` |
+| dense cached | `0.102s` |
+
+For benchmark commands, profiling traces, and methodology, see:
+- `/Users/chriskarani/CodingProjects/Wax/Tasks/hot-path-specialization-investigation.md`
+
 *No, that's not a typo. GPU vector search really is sub-millisecond.*
 
 ---
 
+## WAL Compaction and Storage Health (2026-02)
+
+Wax now includes a WAL/storage health track focused on commit latency tails, long-run file growth, and recovery behavior:
+
+- No-op index compaction guards to avoid unnecessary index rewrites.
+- Single-pass WAL replay with guarded replay snapshot fast path.
+- Proactive WAL-pressure commits for targeted workloads (guarded rollout).
+- Scheduled `rewriteLiveSet` maintenance with dead-payload thresholds, validation, and rollback.
+
+### Measured outcomes
+
+- Repeated unchanged index compaction growth improved from `+61,768,464` bytes over 8 runs (`~7.72MB/run`) to bounded drift (test-gated).
+- Commit latency improved in most matrix workloads in recent runs (examples: `medium_hybrid` p95 `-13.9%`, `large_text_10k` p95 `-8.0%`, `sustained_write_text` p95 `-5.7%`).
+- Reopen/recovery p95 is generally flat-to-improved across the matrix.
+- `sustained_write_hybrid` remains workload-sensitive, so proactive/scheduled maintenance stays guarded by default.
+
+### Safe rollout defaults
+
+- Proactive pressure commits are tuned for targeted workloads and validated with percentile guardrails.
+- Replay snapshot open-path optimization is additive and guarded.
+- Scheduled live-set rewrite is configurable and runs deferred from the `flush()` hot path.
+- Rewrite candidates are automatically validated and rolled back on verification failure.
+
+### Configure scheduled live-set rewrite
+
+```swift
+import Wax
+
+var config = OrchestratorConfig.default
+config.liveSetRewriteSchedule = LiveSetRewriteSchedule(
+    enabled: true,
+    checkEveryFlushes: 32,
+    minDeadPayloadBytes: 64 * 1024 * 1024,
+    minDeadPayloadFraction: 0.25,
+    minimumCompactionGainBytes: 0,
+    minimumIdleMs: 15_000,
+    minIntervalMs: 5 * 60_000,
+    verifyDeep: false
+)
+```
+
+### Reproduce benchmark matrix
+
+```bash
+WAX_BENCHMARK_WAL_COMPACTION=1 \
+WAX_BENCHMARK_WAL_OUTPUT=/tmp/wal-matrix.json \
+swift test --filter WALCompactionBenchmarks.testWALCompactionWorkloadMatrix
+```
+
+```bash
+WAX_BENCHMARK_WAL_GUARDRAILS=1 \
+swift test --filter WALCompactionBenchmarks.testProactivePressureCommitGuardrails
+```
+
+```bash
+WAX_BENCHMARK_WAL_REOPEN_GUARDRAILS=1 \
+swift test --filter WALCompactionBenchmarks.testReplayStateSnapshotGuardrails
+```
+
+See `/Users/chriskarani/CodingProjects/Wax/Tasks/wal-compaction-investigation.md` and `/Users/chriskarani/CodingProjects/Wax/Tasks/wal-compaction-baseline.json` for methodology and full baseline artifacts.
+
+---
+
 ## Quick Start
 
 ### 1. Add to Package.swift
 
 ```swift
-.package(url: "https://github.com/christopherkarani/Wax.git", from: "0.1.1")
+.package(url: "https://github.com/christopherkarani/Wax.git", from: "0.1.6")
 ```
 
 ### 2. Choose Your Memory Type

SHOW_HN_POST.md

@@ -0,0 +1,91 @@
+# Show HN Post
+
+**Title:** `Show HN: Wax -- On-device multimodal RAG for iOS/macOS with Metal GPU search`
+
+**URL:** `https://github.com/christopherkarani/Wax`
+
+---
+
+Hey HN,
+
+I built Wax, an open-source Swift framework for on-device Retrieval-Augmented Generation. It indexes text, photos, and videos into a single portable file and searches them with sub-millisecond latency -- with no server, no API calls, and no data leaving the device.
+
+**Why I built this:** Every RAG solution I found required either a cloud vector database (Pinecone, Weaviate) or a local server process (ChromaDB, Qdrant). I wanted something that works like SQLite -- import the library, open a file, query it. Except for multimodal content with hybrid search.
+
+**What it does:**
+
+- **Single-file storage (`.mv2s`)** -- Everything lives in one crash-safe binary file: embeddings, BM25 index, metadata, compressed payloads. You can sync it via iCloud, email it, or commit it to git. Dual-header atomic writes with generation counters mean you can kill -9 mid-write and never corrupt the database.
+
+- **Metal GPU vector search** -- Vectors live directly in Apple Silicon unified memory (`MTLBuffer`). Zero CPU-GPU copy. Adaptive SIMD4/SIMD8 kernels based on embedding dimensions. GPU-side bitonic sort for top-K. Result: **sub-millisecond search on 10K+ vectors** (vs ~100ms on CPU). Falls back to USearch HNSW on non-Metal hardware.
+
+- **Hybrid search with query-adaptive fusion** -- Four parallel search lanes (BM25, vector, timeline, structured memory) fused with Reciprocal Rank Fusion. A lightweight rule-based classifier detects query intent (factual -> boost BM25, temporal -> boost timeline, semantic -> boost vector). Deterministic tie-breaking means identical queries always produce identical results.
+
+- **Photo RAG** -- Indexes your photo library with OCR, captions, GPS binning (~1km resolution), and per-region embeddings. Query "find that receipt from the restaurant" and it searches OCR text, image similarity, and location simultaneously. Fully offline -- iCloud-only photos get metadata-only indexing (marked as degraded, never silently downloaded).
+
+- **Video RAG** -- Segments videos into configurable time windows, extracts keyframe embeddings, and maps transcripts to segments. Results include timecodes so you can jump to the exact moment. Capture-time semantics: "videos from last week" filters by recording date, not segment position.
+
+- **Deterministic context assembly** -- `FastRAGContextBuilder` produces identical output for identical input under strict token budgets. Three-tier surrogate compression (full/gist/micro) adapts based on memory age and importance. Uses bundled cl100k_base BPE tokenization -- no network, no nondeterminism.
+
+- **Bring your own model** -- Wax ships no ML models by default (optional built-in MiniLM via Swift package trait). You provide embedders, OCR, captions, and transcripts via protocols. Each provider declares `onDeviceOnly` or `networkOptional`, validated at init.
+
+**Technical details:**
+
+- 22K lines of Swift 6.2 (strict concurrency), 496 lines of Metal shaders
+- Every orchestrator is a Swift actor -- thread safety proven at compile time
+- Custom binary codec (little-endian, deterministic serialization, SHA256 checksums)
+- Two-phase indexing: stage to WAL, commit atomically
+- 91 test files covering integration, property-based, and stress scenarios
+- iOS 26+ / macOS 26+
+
+**Quick start:**
+
+```swift
+import Wax
+
+let brain = try await MemoryOrchestrator(
+    at: URL(fileURLWithPath: "brain.mv2s")
+)
+
+// Remember
+try await brain.remember(
+    "User prefers dark mode and gets headaches from bright screens",
+    metadata: ["source": "onboarding"]
+)
+
+// Recall with RAG
+let context = try await brain.recall(query: "user preferences")
+for item in context.items {
+    print("[\(item.kind)] \(item.text)")
+}
+```
+
+For more control, the low-level API exposes the full storage engine:
+
+```swift
+import Wax
+import WaxCore
+
+let store = try await Wax.create(at: fileURL)
+let session = try await WaxSession(wax: store, mode: .readWrite())
+
+let content = Data("Meeting notes from Q4 planning...".utf8)
+try await session.put(content, options: FrameMetaSubset(
+    kind: "note.meeting",
+    searchText: "Meeting notes from Q4 planning...",
+    metadata: Metadata(["date": "2026-01-15"])
+))
+try await session.commit()
+
+let response = try await session.search(
+    SearchRequest(query: "Q4 planning decisions", topK: 5)
+)
+```
+
+**What it's not:**
+- Not a cloud service. No telemetry. No vendor lock-in.
+- Not an LLM. Wax retrieves context for your LLM of choice.
+- Not Python. This is native Swift, optimized for Apple Silicon.
+
+Feedback welcome. The framework is early but the core architecture (storage format, search pipeline, concurrency model) is stable.
+
+GitHub: https://github.com/christopherkarani/Wax

Sources/Wax/Adapters/CLAUDE.md

@@ -0,0 +1,7 @@
+<claude-mem-context>
+# Recent Activity
+
+<!-- This section is auto-generated by claude-mem. Edit content outside the tags. -->
+
+*No recent activity*
+</claude-mem-context>

Sources/Wax/Embeddings/CLAUDE.md

@@ -0,0 +1,7 @@
+<claude-mem-context>
+# Recent Activity
+
+<!-- This section is auto-generated by claude-mem. Edit content outside the tags. -->
+
+*No recent activity*
+</claude-mem-context>

Sources/Wax/Embeddings/EmbeddingMemoizer.swift

@@ -160,6 +160,13 @@ actor EmbeddingMemoizer {
     }
 }
 
+extension EmbeddingMemoizer {
+    static func fromConfig(capacity: Int, enabled: Bool = true) -> EmbeddingMemoizer? {
+        guard enabled, capacity > 0 else { return nil }
+        return EmbeddingMemoizer(capacity: capacity)
+    }
+}
+
 enum EmbeddingKey {
     static func make(text: String, identity: EmbeddingIdentity?, dimensions: Int, normalized: Bool) -> UInt64 {
         var hasher = FNV1a64()

Sources/Wax/Ingest/CLAUDE.md

@@ -33,3 +33,12 @@ All PDF code is wrapped in `#if canImport(PDFKit)`. This means:
 - `PDFKit` (Apple platforms only)
 - `Foundation`
 - `MemoryOrchestrator` (from parent Wax module)
+
+
+<claude-mem-context>
+# Recent Activity
+
+<!-- This section is auto-generated by claude-mem. Edit content outside the tags. -->
+
+*No recent activity*
+</claude-mem-context>

Sources/Wax/Ingest/FileIngestError.swift

@@ -0,0 +1,24 @@
+import Foundation
+
+/// Errors that can occur while ingesting a local text file.
+public enum FileIngestError: Error, Sendable, Equatable {
+    case fileNotFound(url: URL)
+    case loadFailed(url: URL)
+    case unsupportedTextEncoding(url: URL)
+    case emptyContent(url: URL)
+}
+
+extension FileIngestError: LocalizedError {
+    public var errorDescription: String? {
+        switch self {
+        case let .fileNotFound(url):
+            return "File not found: \(url.path)"
+        case let .loadFailed(url):
+            return "File could not be read: \(url.path)"
+        case let .unsupportedTextEncoding(url):
+            return "File is not UTF-8 text: \(url.path)"
+        case let .emptyContent(url):
+            return "File has no text content: \(url.path)"
+        }
+    }
+}

Sources/Wax/Ingest/TextChunker.swift

@@ -22,7 +22,17 @@ public enum TextChunker {
         let cappedTarget = max(1, targetTokens)
         let cappedOverlap = max(0, overlapTokens)
 
-        guard let counter = try? await TokenCounter.shared() else { return [text] }
+        let counter: TokenCounter
+        do {
+            counter = try await TokenCounter.shared()
+        } catch {
+            WaxDiagnostics.logSwallowed(
+                error,
+                context: "text chunker token counter init",
+                fallback: "character-preserving unsplit text"
+            )
+            return [text]
+        }
         let tokens = await counter.encode(text)
         if tokens.count <= cappedTarget {
             return [text]
@@ -58,7 +68,15 @@ public enum TextChunker {
 
         return AsyncStream { continuation in
             Task {
-                guard let counter = try? await TokenCounter.shared() else {
+                let counter: TokenCounter
+                do {
+                    counter = try await TokenCounter.shared()
+                } catch {
+                    WaxDiagnostics.logSwallowed(
+                        error,
+                        context: "text chunker stream token counter init",
+                        fallback: "stream original unsplit text"
+                    )
                     continuation.yield(text)
                     continuation.finish()
                     return

Sources/Wax/Maintenance/CLAUDE.md

@@ -0,0 +1,7 @@
+<claude-mem-context>
+# Recent Activity
+
+<!-- This section is auto-generated by claude-mem. Edit content outside the tags. -->
+
+*No recent activity*
+</claude-mem-context>

Sources/Wax/Maintenance/LiveSetRewriteOptions.swift

@@ -0,0 +1,22 @@
+import Foundation
+
+public struct LiveSetRewriteOptions: Sendable, Equatable {
+    /// Allow replacing an existing destination file.
+    public var overwriteDestination: Bool
+
+    /// Replace payload bytes for non-live frames (deleted/superseded) with empty payloads.
+    public var dropNonLivePayloads: Bool
+
+    /// Run `Wax.verify(deep:)` on the rewritten file before returning.
+    public var verifyDeep: Bool
+
+    public init(
+        overwriteDestination: Bool = false,
+        dropNonLivePayloads: Bool = true,
+        verifyDeep: Bool = false
+    ) {
+        self.overwriteDestination = overwriteDestination
+        self.dropNonLivePayloads = dropNonLivePayloads
+        self.verifyDeep = verifyDeep
+    }
+}