docs: fix confidence scoring accuracy and default threshold (#132)

* docs: add confidence scoring explanation page

* docs: fix confidence scoring doc accuracy and default threshold value

---------

Co-authored-by: ahfoysal <ahfoysal30@gmail.com>

Author: rajkumarsakthivel

docs/wiki/Confidence-Scoring.md

@@ -0,0 +1,61 @@
+# Confidence Scoring
+
+CCE ranks every retrieved chunk with a confidence score before returning it.
+
+## What the score combines
+
+1. **Vector similarity score (50%)**
+   - Uses cosine distance from the embedding query match.
+   - The raw distance is normalized to `[0, 1]` by dividing by 2, then converted: `max(0, 1 - normalized_distance)`.
+
+2. **Keyword/file-hint score (40%)**
+   - Uses parser signal such as matched keywords and file hints.
+   - A distance in `[0, 5]` is converted to a score: `max(0, 1 - keyword_distance / 5)`.
+
+3. **Recency score (10%)**
+   - Newer chunks get higher weight using exponential decay.
+   - Missing `modified_ts` metadata defaults to a neutral score of `0.5`.
+   - Half-life is one week.
+
+## Formula
+
+The confidence value is a weighted sum:
+
+```python
+confidence = (0.5 * vector_score) + (0.4 * keyword_score) + (0.1 * recency_score)
+```
+
+Clamped to `[0.0, 1.0]`.
+
+## Final ranking stages
+
+The confidence value is blended with the hybrid retriever signal (RRF) before filtering:
+
+- Hybrid vector + full-text scores are merged first (RRF), then normalized to `[0, 1]`.
+- Each chunk gets a final score that is a 50/50 blend: `0.5 * confidence + 0.5 * normalized_rrf`.
+- Path penalties are applied (test files and docs are down-weighted by 20%).
+- Chunks below `confidence_threshold` are dropped.
+- Remaining chunks are sorted high-to-low by final score.
+
+Note: `confidence_threshold` is compared against the blended final score, not the raw confidence value. A chunk with a strong RRF rank can pass the threshold even if its raw confidence score is modest.
+
+A higher score means the chunk is considered more relevant and trustworthy for that query.
+
+## `confidence_threshold`
+
+Configured under `retrieval.confidence_threshold` in `~/.cce/config.yaml`:
+
+- Lower values (for example `0.1`) return more results.
+- Higher values (for example `0.7`) return fewer but tighter matches.
+- Default is `0.2`.
+
+```yaml
+retrieval:
+  confidence_threshold: 0.2
+```
+
+## Practical tuning
+
+- If your queries feel too narrow, lower the threshold slightly (for example `0.1`).
+- If you want cleaner, fewer results, raise it (for example `0.5`).
+- The value is compared against the blended score (confidence + RRF), so it controls both relevance and retrieval rank.

docs/wiki/Configuration.md

@@ -31,7 +31,7 @@ indexer:
 
 retrieval:
   top_k: 20              # Maximum number of chunks to return per query
-  confidence_threshold: 0.5  # Minimum confidence score to include a result (0.0–1.0)
+  confidence_threshold: 0.2  # Minimum confidence score to include a result (0.0–1.0)
 
 embedding:
   model: BAAI/bge-small-en-v1.5  # Embedding model (fastembed-compatible)
@@ -116,7 +116,7 @@ You do not need to set this manually — it is detected at startup.
 
 **`top_k`** — how many chunks the retriever returns per query. Higher values surface more context but cost more tokens. Default: 20.
 
-**`confidence_threshold`** — minimum score to include a result. Range 0.0 to 1.0. Lower values return more results; higher values return only strong matches. Default: 0.5.
+**`confidence_threshold`** — minimum score to include a result. Range 0.0 to 1.0. Lower values return more results; higher values return only strong matches. Default: 0.2.
 
 At runtime, Claude can pass `top_k` and `max_tokens` directly to `context_search`:
 ```

docs/wiki/How-It-Works.md

@@ -88,6 +88,8 @@ Every chunk gets a final confidence score combining:
 
 Only chunks above the configured `confidence_threshold` (default 0.5) are returned.
 
+For the exact formula and tuning notes, see [Confidence Scoring](Confidence-Scoring.md).
+
 ---
 
 ## 6. Compression