feat(semantic): [alpha build] provider-aware typed embeddings, reranking, diagnostics, and eval harness

[Zireael]

Summary

Semantic search in AFT moves from a minimal embedding-and-cosine prototype to a provider-capability-aware retrieval subsystem with typed vectors, optional reranking, background lifecycle management, diagnostics, and evaluation tooling. This is a public preview — the feature is functional and tested (~93 new tests) but expects iteration based on real-world feedback.

What changed

The upgrade touches the full semantic pipeline — config, indexing, retrieval, diagnostics, and observability — without breaking the default fastembed experience.

Typed vector representations

Vectors are no longer opaque f32 blobs. Every stored vector carries explicit type metadata (DenseF32, Int8SourceDecoded, BinaryPacked) and is paired with its source kind so the correct distance metric is selected automatically. Binary packed vectors use Hamming search (native bitwise XOR + popcount) instead of cosine, which is both faster and semantically correct for quantized embeddings. This unlocks Perplexity's base64_binary and base64_int8 output modes alongside standard dense providers.

Provider capability profiles

Each embedding backend (fastembed, OpenAI-compatible, Ollama, Perplexity) declares what it supports: output encoding, distance metric, dimension range, max batch size. The config layer validates combinations at configure time — you cannot accidentally request binary vectors through a cosine-only provider. Profiles also carry fingerprint fields so switching providers triggers a clean index rebuild rather than silent corruption.

Fingerprint-driven index lifecycle

A SemanticIndexFingerprint captures every dimension that affects index correctness: backend, model, base_url, dimension, chunking_version, output_encoding, storage_strategy, vector kinds, normalization, and prompt hashes. diff() classifies changes as Rebuild (structural — re-embed everything), ClearQueryCache (query prompts changed — invalidate cached results only), or None. This replaces the previous "delete and hope" invalidation with precise, explainable rebuild decisions.

Non-blocking cold start

Index builds run in a background thread with cooperative cancellation (SemanticCancellationToken via AtomicU64 generation counter). The build checks the generation before each embedding batch and exits early when a reconfigure arrives. Priority ordering ensures high-value files (recently edited, high PageRank) get embedded first. Exponential backoff handles transient provider failures without blocking the session.

Stale-vector pruning

When files are edited, deleted, moved, excluded, or re-included, the index tracks which vectors are stale and prunes them during the next refresh cycle. Every vector record carries file/chunk ownership metadata (file path, version, chunk hash, index fingerprint) so pruning is traceable and deterministic.

File policy and docs chunking

A configurable file policy controls which files enter the index (include globs, exclude globs, max file size, max chunk count). The docs chunker splits Markdown and documentation files into semantic sections before embedding, improving recall for documentation-shaped queries.

Reranking pipeline

Optional reranking via any OpenAI-compatible /v1/rerank or chat-completion endpoint. The pipeline sends initial retrieval candidates to a reranker, parses the response (supporting multiple JSON shapes), and reorders results with safe fallback — if the reranker fails, the original cosine-similarity order is returned unchanged. Config fields: rerank.enabled, rerank.model, rerank.base_url, rerank.api_key_env, rerank.max_candidates.

Search pipeline metrics and diagnostics

Every aft_search call records timing, cache hits/misses, result counts, and reranker fallback events. Metrics are exposed through the status command and through JSONL diagnostic logs for offline analysis. The DiagnosticsOutputMode config controls verbosity in tool output (compact | verbose | off).

Semantic doctor

semantic_doctor is a health-check command that reports config summary, index summary, metrics summary, provider summary, and actionable suggestions. Use it to verify that the index is healthy, the provider is reachable, and the configuration is consistent.

Semantic eval harness

semantic_eval runs a JSONL-defined evaluation suite against the semantic index. Each case specifies a query, expected paths, expected symbols, and top-k. The harness computes recall@k and MRR (Mean Reciprocal Rank) for quantifying retrieval quality across config changes.

Status integration

The status command now includes semantic health metrics: lifecycle state, entry count, dimension, total queries, cache hit ratio, average query time, and provider info. The OpenCode TUI sidebar surfaces these alongside the existing index state.

Config trust boundary

backend, base_url, and api_key_env are user-only fields — project-level aft.jsonc cannot inject these. A hostile repository cannot redirect embeddings at an attacker-controlled endpoint or exfiltrate API keys. The plugin logs a warning when it strips a project-level setting.

Contextualized document-chunk embedding (partial)

Initial support for Perplexity-style document/chunk grouped embedding — chunks from the same source document are batched together rather than flattened. Oversized document handling and retry logic are still in progress (see roadmap).

How to test

Default fastembed (zero-config)

# Enable semantic search in your AFT config
# ~/.config/opencode/aft.jsonc or ~/.pi/agent/aft.jsonc:
{ "semantic_search": true }

# Start a session — index builds in background
# Run aft_search with a concept query:
aft_search({ "query": "authentication middleware" })

Verify: results appear with source: semantic or source: hybrid tags. Status shows [index: ready] after build completes.

Provider switching

// Switch to OpenAI-compatible
{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  }
}

Verify: index rebuilds automatically on next session start. Status shows new provider/model.

Reranking

{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  },
  "rerank": {
    "enabled": true,
    "model": "rerank-english-v3.0",
    "base_url": "https://api.cohere.com",
    "api_key_env": "COHERE_API_KEY"
  }
}

Verify: search results show reranker-sorted order. Disable reranker — results fall back to cosine order.

Semantic doctor

aft_search({ "query": "test" })  # trigger index build if cold
# Then check health via status command or semantic_doctor

Verify: health report shows ConfigSummary, IndexSummary, MetricsSummary, ProviderSummary.

Eval harness

// Create eval-cases.jsonl:
{"query": "authentication handler", "expected_paths": ["src/auth/middleware.ts"], "expected_symbols": ["authMiddleware"], "top_k": 10}
{"query": "database connection", "expected_paths": ["src/db/pool.ts"], "expected_symbols": ["createPool"], "top_k": 10}

Verify: returns recall@k and MRR scores.

Test coverage

~93 tests across 8 test sub-tasks covering:

• Config parsing and backward compatibility
• Fingerprint diff matrix (all field combinations → Rebuild/ClearQueryCache/None)
• File policy, docs chunking, and manifest handling
• VectorStore trait with DenseF32 and BinaryPacked implementations
• Binary packed-vector storage and Hamming search
• Lifecycle states, snapshots, and stale-vector pruning
• Search pipeline metrics, diagnostics, and DiagnosticsOutputMode
• Concurrency, race conditions, and cancellation token behavior
• Security trust boundary enforcement (project config stripping)
• Semantic doctor health report
• Semantic eval harness (JSONL parsing, scoring, recall/MRR)
• Reranking pipeline (parse multiple JSON shapes, fallback on failure)

Roadmap

Still in progress or planned for follow-up:

• aft-t6p.23: Complete contextualized document-chunk embedding (oversized docs, retry logic) — partially implemented
• aft-t6p.2.2: Configurable snippet truncation in reranking (currently hardcoded at 200 chars)
• aft-t6p.18: End-to-end verification across all backends
• aft-t6p.5: Configuration and operations documentation
• Performance benchmarking suite
• Migration tooling for index format upgrades

Architecture notes

Key new modules:

• crates/aft/src/semantic_rerank.rs — reranking pipeline with safe fallback
• crates/aft/src/semantic_diagnostics.rs — JSONL diagnostic logging
• crates/aft/src/semantic_doctor.rs — health-check report generation
• crates/aft/src/semantic_eval.rs — evaluation harness (JSONL parser, scoring)
• crates/aft/src/vector_store.rs — VectorStore trait with DenseF32 and BinaryPacked implementations
• crates/aft/src/commands/semantic_doctor.rs — doctor command handler
• crates/aft/src/commands/semantic_eval.rs — eval command handler

Modified significantly:

• crates/aft/src/semantic_index.rs — lifecycle management, fingerprint-driven invalidation, non-blocking build, stale pruning, typed vectors
• crates/aft/src/config.rs — provider profiles, rerank config, trust boundary fields
• crates/aft/src/commands/status.rs — semantic health metrics
• crates/aft/src/commands/semantic_search.rs — reranking integration, diagnostics output mode

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Upgrades semantic search to a provider-aware pipeline with typed vectors, reranking, contextualized document-chunk embedding, partial-ready querying, and built-in diagnostics/eval. Adds Perplexity support and Hamming search for binary/int8, and hardens lifecycle, metrics, and config.

• New Features

  • Provider profiles and typed vectors (f32, int8, binary packed) with auto metric selection; enables Perplexity base64_binary/base64_int8.
  • Contextualized document-chunk embedding with oversized-document splitting, retry/backoff, and build diagnostics.
  • Fingerprint-driven lifecycle with background builds, partial-ready state, and precise stale‑vector pruning.
  • Optional reranking via OpenAI-compatible endpoints with safe fallback.
  • Metrics/diagnostics: JSONL logs with retention and configurable verbosity; status, semantic_doctor, and semantic_eval surface semantic health.

• Bug Fixes

  • Cosine similarity guards zero-norm vectors and clamps scores to [-1, 1].
  • Reranker: robust fence stripping, out-of-bounds index warning, duplicate index prevention, and max_candidate_chars support.
  • Search no longer overwrites lifecycle to Ready; validates non-empty queries; cancellation token uses acquire/release ordering.
  • Chunking fixes: correct end_line for final splits and accurate oversized-document counters.
  • Config parsing accepts all semantic fields (dimensions, encoding, storage strategy, input mode, metric, prompts, diagnostics, rerank limits) and now also reads jsonl_logging, jsonl_path, include_raw_queries, include_snippets, retention_days, and metrics_window_size; TypeScript enums aligned with Rust.

^{Written for commit d204e2d. Summary will update on new commits.}

Greptile Summary

This PR upgrades the semantic search subsystem from a minimal prototype to a full provider-aware retrieval pipeline, introducing typed vectors, fingerprint-driven index lifecycle management, optional reranking, background build with cooperative cancellation, and diagnostic tooling.

• Core pipeline (semantic_index.rs, vector_store.rs): adds VectorStore abstraction, EmbeddingModelProfile for provider capability validation, SemanticIndexFingerprint with diff() for precise rebuild decisions, and stale-vector pruning. Cancellation token uses correct Acquire/Release ordering; cosine_similarity guards zero-norm vectors and clamps output.
• Reranking (semantic_rerank.rs, semantic_search.rs): optional OpenAI-compatible reranker with safe fallback and markdown-fence stripping. A field/method naming collision on diagnostics_enabled silently disables JSONL logging when jsonl_logging: true is set without diagnostics_enabled: true.
• TypeScript config (config.ts): new enum schemas and trust-boundary mergeSemanticConfig; rerank and diagnostics fields are absent from SemanticConfigSchema and stripped by Zod before reaching Rust.

Confidence Score: 3/5

Safe to merge with the diagnostics_enabled field/method fix applied — without it, any user who enables JSONL logging alone gets silence.

The search handler reads the raw bool field instead of the diagnostics_enabled() method that unifies diagnostics_enabled || jsonl_logging, silently breaking JSONL-only logging configs.

crates/aft/src/commands/semantic_search.rs (line 51 field vs. method) and packages/opencode-plugin/src/config.ts (rerank/diagnostics fields absent from SemanticConfigSchema).

Important Files Changed

Filename	Overview
crates/aft/src/commands/semantic_search.rs	Rewrites search handler to add reranking, diagnostics, and partial-index handling. Field access instead of method call on diagnostics_enabled silently disables JSONL logging when only jsonl_logging: true is set.
crates/aft/src/semantic_rerank.rs	New reranking pipeline with safe fallback. Markdown-fence stripper has an edge case for single-line format that returns empty string, causing a graceful but silent degradation.
crates/aft/src/config.rs	Adds provider enums, SemanticBackendConfig with rerank/diagnostics fields, and a diagnostics_enabled() method that ORs the field with jsonl_logging — but callers bypass this method and read the field directly.
packages/opencode-plugin/src/config.ts	Adds SemanticConfigSchema with new enum types and project-level trust boundary enforcement. Missing rerank/diagnostics fields from the schema (silently stripped by Zod) and minor warning message formatting inconsistency.
crates/aft/src/vector_store.rs	New VectorStore trait + FlatF32 and FlatBinaryHamming implementations. Search, upsert, prune, and orphan-removal logic is correct and well-tested.
crates/aft/src/semantic_index.rs	Major expansion adding typed vectors, fingerprint-driven lifecycle, background build with cooperative cancellation, stale-vector pruning, and provider profiles. Cosine similarity correctly guards zero-norm vectors and clamps output.
crates/aft/src/semantic_diagnostics.rs	JSONL diagnostics logger and in-memory metrics. Correct use of retention policy and configurable verbosity. No issues found.
crates/aft/src/context.rs	Adds SemanticCancellationToken (AtomicU64 with correct Acquire/Release ordering) and lazy JSONL logger initialization. Logic in init_diagnostics_logger is correct, gated on jsonl_logging field.

Comments Outside Diff (1)

1. packages/opencode-plugin/src/config.ts, line 37-54 (link)

   TypeScript enum values don't match the Rust serde strings — config will fail to deserialize

   Several new enum schemas use values that don't align with the Rust serde representation:

   • SemanticOutputEncodingEnum allows "binary", "ubinary", "int8", "uint8" but Rust OutputEncoding deserializes from "base64_binary" and "base64_int8".
   • SemanticStorageStrategyEnum allows "flat" and "binary_pack" but Rust StorageStrategy expects "native_f32" and "binary_packed".
   • SemanticInputModeEnum includes "chunk_extracts" and "contextualized" but Rust InputMode only has "flat_texts" and "document_chunks".
   • SemanticDistanceMetricEnum uses "dot" but Rust DistanceMetric expects "dot_product".
   • SemanticBackendEnum is missing the new "perplexity" variant added to Rust.

   A user who follows the TypeScript autocomplete and picks output_encoding: "int8" will pass TypeScript validation but receive a deserialization error (or silent fallback to default) from the Rust binary at runtime.

_{Reviews (5): Last reviewed commit: "fix(configure): add missing JSONL/metric..." | Re-trigger Greptile}

[cubic-dev-ai]

4 issues found across 107 files

<sub>Note: This PR contains a large number of files. cubic only reviews up to 100 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 69 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".gitignore">

<violation number="1" location=".gitignore:95">
P2: Inconsistent .gitignore pattern: `omo/` should likely be `.omo/` to match the hidden tooling directory convention used by all other entries in this block.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Inconsistent .gitignore pattern: omo/ should likely be .omo/ to match the hidden tooling directory convention used by all other entries in this block.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .gitignore, line 95:

<comment>Inconsistent .gitignore pattern: `omo/` should likely be `.omo/` to match the hidden tooling directory convention used by all other entries in this block.</comment>

<file context>
@@ -87,3 +87,11 @@ benchmarks/aft-search/.bench/
+.beads/
+.qartez/
+.claude/
+omo/
+.kiro/
+.lean-ctx/
</file context>

[Zireael]

Source code for semantic search functionality for public preview.
Feature skeleton is there, needs finishing up, polishing static tests and functional testing.
One more thing that would need adding would be model2vec 'Potion Code 16M' support. If it performs well in tests, I think it could become fast, cheap and performant default semantic model.

Here's imlementation plans for sprints under this epic (in gastown beads format):
aft-semantic-search-upgrade.json

[cubic-dev-ai]

1 issue found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode-plugin/src/config.ts">

<violation number="1" location="packages/opencode-plugin/src/config.ts:40">
P2: Semantic enum literals were renamed without backward-compatibility aliases or migration, breaking existing configs that use old values.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Semantic enum literals were renamed without backward-compatibility aliases or migration, breaking existing configs that use old values.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode-plugin/src/config.ts, line 40:

<comment>Semantic enum literals were renamed without backward-compatibility aliases or migration, breaking existing configs that use old values.</comment>

<file context>
@@ -34,19 +34,19 @@ const CheckerEnum = z.enum([
 
 /** Output encoding mode for embeddings. */
-const SemanticOutputEncodingEnum = z.enum(["float", "binary", "ubinary", "int8", "uint8"]);
+const SemanticOutputEncodingEnum = z.enum(["float", "base64_int8", "base64_binary"]);
 
 /** Storage strategy for embedding vectors. */
</file context>

[cubic-dev-ai]

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/aft/src/commands/configure.rs">

<violation number="1" location="crates/aft/src/commands/configure.rs:342">
P1: `rerank_base_url` is parsed without SSRF validation, unlike `base_url`</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: rerank_base_url is parsed without SSRF validation, unlike base_url

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At crates/aft/src/commands/configure.rs, line 342:

<comment>`rerank_base_url` is parsed without SSRF validation, unlike `base_url`</comment>

<file context>
@@ -230,6 +230,150 @@ fn parse_semantic_config(
+            .to_string()
+            .into();
+    }
+    if let Some(raw) = obj.get("rerank_base_url") {
+        semantic.rerank_base_url = raw
+            .as_str()
</file context>

[cubic-dev-ai]

2 issues found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/aft/src/commands/configure.rs">

<violation number="1" location="crates/aft/src/commands/configure.rs:382">
P2: `semantic.jsonl_path` lacks path validation/normalization, unlike other path configs in this file (`validate_storage_dir`, `parse_lsp_paths_extra`) which enforce absolute paths and reject `..` traversal. This creates path-injection risk for downstream JSONL diagnostics writes.</violation>

<violation number="2" location="crates/aft/src/commands/configure.rs:402">
P2: `semantic.retention_days` uses lossy `u64 -> u32` cast with silent overflow instead of explicit validation.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: semantic.jsonl_path lacks path validation/normalization, unlike other path configs in this file (validate_storage_dir, parse_lsp_paths_extra) which enforce absolute paths and reject .. traversal. This creates path-injection risk for downstream JSONL diagnostics writes.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At crates/aft/src/commands/configure.rs, line 382:

<comment>`semantic.jsonl_path` lacks path validation/normalization, unlike other path configs in this file (`validate_storage_dir`, `parse_lsp_paths_extra`) which enforce absolute paths and reject `..` traversal. This creates path-injection risk for downstream JSONL diagnostics writes.</comment>

<file context>
@@ -374,6 +374,42 @@ fn parse_semantic_config(
+            "configure: semantic.jsonl_logging must be a boolean".to_string()
+        })?;
+    }
+    if let Some(raw) = obj.get("jsonl_path") {
+        semantic.jsonl_path = if raw.is_null() {
+            None
</file context>

[cubic-dev-ai]

P2: semantic.retention_days uses lossy u64 -> u32 cast with silent overflow instead of explicit validation.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At crates/aft/src/commands/configure.rs, line 402:

<comment>`semantic.retention_days` uses lossy `u64 -> u32` cast with silent overflow instead of explicit validation.</comment>

<file context>
@@ -374,6 +374,42 @@ fn parse_semantic_config(
+        })?;
+    }
+    if let Some(raw) = obj.get("retention_days") {
+        semantic.retention_days = raw.as_u64().ok_or_else(|| {
+            "configure: semantic.retention_days must be an unsigned integer".to_string()
+        })? as u32;
</file context>

Suggested change

	semantic.retention_days = raw.as_u64().ok_or_else(|| {
	"configure: semantic.retention_days must be an unsigned integer".to_string()
	})? as u32;
	let v = raw.as_u64().ok_or_else(|| {
	"configure: semantic.retention_days must be an unsigned integer".to_string()
	})?;
	semantic.retention_days = u32::try_from(v)
	.map_err(|_| "configure: semantic.retention_days is too large".to_string())?;