Skip to content

Commit 387dea0

Browse files
docs: polish sweep — fix stale docstrings, dead paths, PyPI-visible links (closes #21, #22, #23, #24, #25, #26, #27, #15)
Hunter Bundle E. 8 findings that mostly didn't affect runtime but degraded the first-read experience — wrong tier copy, broken links on PyPI, stale PNG duplicates, gitignored paths cited as sources. - F12 (#21): vector_search.py module docstring rewritten — acknowledges tier-aware embedder resolution (Model2Vec / Qwen3 per tier), not the v0.3.0 Model2Vec-only framing. Adds sentence-transformers to Dependencies. - F13 (#22): bench_truememory_base.py header — was copy-pasted from Pro and still said "Pro Tier (+HyDE)". Rewritten to describe Base (Qwen3 + gte-reranker, HyDE off, 91.5%). Usage examples now reference the right script name. - F14 (#23): category-heatmap.png deleted — was byte-identical to category-grouped-bars.png (MD5 f00c08...). Only live reference was to grouped-bars; heatmap filename was dead (CHANGELOG historical mention only). - F15 (#24): README v0.3.0-chart caveat moved above the first benchmark chart (was after all three). Option 2 interim per the finding — chart regeneration (Option 1) is a separate deferred task. - F16 (#25): BENCHMARK_RESULTS.md no longer cites gitignored `_working/benchmarks_56combo/...` paths as authoritative. The aggregate numbers in the table are the canonical reference; the Modal scripts reproduce them. - F17 (#26): 8 relative text links in README rewritten to absolute github.com/.../blob/main/... URLs. PyPI's markdown renderer doesn't rewrite relative paths, so the links 404'd on the package page for install.sh, CLAUDE.md.example, LICENSE, and four benchmarks/* refs. - F18 (#27): `_nm_format_ctx` / `_nm_make_hyde_fn` renamed to `_tm_*` in the four bench scripts (bench_truememory_{edge,base,pro}.py, modal_benchmark.py) — stray neuromem-namespace helpers from pre-rebrand. Purely cosmetic; scripts are ship-in-sdist so the brand leak was visible. - F40 (#15): documented `--help` precedence over `--setup` (Unix convention — docs-emitting flags short-circuit side-effects) with an inline comment in mcp_server.main and a regression-lock test in test_cli_help.py. pytest 197 passed / 1 skipped (baseline 196 + 1 new F40 test). ruff clean. Local `python -m build && twine check --strict` PASSED.
1 parent b1c29bf commit 387dea0

10 files changed

Lines changed: 72 additions & 38 deletions

File tree

README.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@
2929

3030
Tested on [LoCoMo](https://github.com/snap-research/locomo), the standard benchmark for conversational memory. 1,540 questions across 10 conversations. All 8 systems share the same answer model, judge, scoring, top-k, and byte-identical answer prompt — only retrieval differs.
3131

32+
> **Note on charts:** the hero banner and the three charts below still show the v0.3.0 single-Pro-tier layout (91.5%). Chart regeneration is tracked for a later release; all numerical claims in the README text and tables reflect the v0.4.0 three-tier scores (90.1 / 91.5 / 91.8%).
33+
3234
<p align="center">
3335
<img src="https://raw.githubusercontent.com/buildingjoshbetter/TrueMemory/main/assets/charts/leaderboard-bar.png?v=2" alt="LoCoMo 8-System Comparison" />
3436
</p>
@@ -39,7 +41,7 @@ TrueMemory achieves **state-of-the-art accuracy for fully-local memory systems**
3941
<img src="https://raw.githubusercontent.com/buildingjoshbetter/TrueMemory/main/assets/charts/accuracy-vs-cost.png?v=2" alt="Accuracy vs Infrastructure Cost" />
4042
</p>
4143

42-
All scores use the same evaluation pipeline: GPT-4.1-mini answer generation, GPT-4o-mini judge (3x majority vote), temperature=0. Zero errors across 12,320 total answers. Scores use a lenient semantic-match judge; rankings are valid across all systems but absolute values are higher than published LoCoMo baselines using strict exact-match. [Full methodology](benchmarks/locomo/BENCHMARK_RESULTS.md) and reproduction scripts in [`benchmarks/`](benchmarks/locomo/).
44+
All scores use the same evaluation pipeline: GPT-4.1-mini answer generation, GPT-4o-mini judge (3x majority vote), temperature=0. Zero errors across 12,320 total answers. Scores use a lenient semantic-match judge; rankings are valid across all systems but absolute values are higher than published LoCoMo baselines using strict exact-match. [Full methodology](https://github.com/buildingjoshbetter/TrueMemory/blob/main/benchmarks/locomo/BENCHMARK_RESULTS.md) and reproduction scripts in [`benchmarks/`](https://github.com/buildingjoshbetter/TrueMemory/blob/main/benchmarks/locomo/).
4345

4446
---
4547

@@ -57,8 +59,6 @@ All scores use the same evaluation pipeline: GPT-4.1-mini answer generation, GPT
5759

5860
TrueMemory Pro nearly matches EverMemOS across all 4 question categories. Mem0 collapses on multi-hop reasoning (37.7% vs 90.7%).
5961

60-
> The hero banner and charts in this README still show the v0.3.0 single-Pro-tier layout (91.5%). Chart regeneration is tracked for a later release; all numerical claims in the README text and tables reflect the v0.4.0 three-tier scores above.
61-
6262
---
6363

6464
## 🏗️ Edge / Base / Pro
@@ -101,7 +101,7 @@ curl -LsSf https://raw.githubusercontent.com/buildingjoshbetter/TrueMemory/main/
101101

102102
> **What this actually does:** installs [uv](https://docs.astral.sh/uv/) (Astral's Python tool manager) if needed, fetches a managed Python 3.12 into `~/.local/share/uv/`, installs TrueMemory into an isolated tool environment, and auto-configures Claude Code and Claude Desktop. **Your system Python is never touched.** No sudo, no venvs, no pip struggle. Uninstall cleanly with `uv tool uninstall truememory`.
103103
104-
> **Want to audit the script first?** It's ~140 lines of shell, no sudo, stays entirely under `$HOME`. Read the source at [`install.sh`](install.sh), or download and inspect locally: `curl -LsSf https://raw.githubusercontent.com/buildingjoshbetter/TrueMemory/main/install.sh -o install.sh && less install.sh && sh install.sh`.
104+
> **Want to audit the script first?** It's ~140 lines of shell, no sudo, stays entirely under `$HOME`. Read the source at [`install.sh`](https://github.com/buildingjoshbetter/TrueMemory/blob/main/install.sh), or download and inspect locally: `curl -LsSf https://raw.githubusercontent.com/buildingjoshbetter/TrueMemory/main/install.sh -o install.sh && less install.sh && sh install.sh`.
105105
106106
> **Want Base or Pro (adds Qwen3 embeddings + gte-reranker + sentence-transformers, ~1.5-2.5GB depending on OS)?**
107107
> ```bash
@@ -149,7 +149,7 @@ After setup, TrueMemory runs automatically. It stores what you tell it and recal
149149
150150
### Make it automatic (optional)
151151
152-
Copy [`CLAUDE.md.example`](CLAUDE.md.example) to your home directory as `CLAUDE.md`. This tells Claude to store your preferences and recall them without being asked:
152+
Copy [`CLAUDE.md.example`](https://github.com/buildingjoshbetter/TrueMemory/blob/main/CLAUDE.md.example) to your home directory as `CLAUDE.md`. This tells Claude to store your preferences and recall them without being asked:
153153
154154
```bash
155155
cp CLAUDE.md.example ~/CLAUDE.md
@@ -214,9 +214,9 @@ uvx creates a cached environment on first run; subsequent spawns are fast. Good
214214
215215
Every benchmark script is self-contained and runs on [Modal](https://modal.com).
216216
217-
- **[Leaderboard & Reproduction](benchmarks/locomo/README.md)**: run any system yourself
218-
- **[Full Technical Report](benchmarks/locomo/BENCHMARK_RESULTS.md)**: per-category breakdowns, latency, cost, hardware
219-
- **[Evaluation Config](benchmarks/locomo/EVAL_CONFIG.md)**: exact models, prompts, parameters
217+
- **[Leaderboard & Reproduction](https://github.com/buildingjoshbetter/TrueMemory/blob/main/benchmarks/locomo/README.md)**: run any system yourself
218+
- **[Full Technical Report](https://github.com/buildingjoshbetter/TrueMemory/blob/main/benchmarks/locomo/BENCHMARK_RESULTS.md)**: per-category breakdowns, latency, cost, hardware
219+
- **[Evaluation Config](https://github.com/buildingjoshbetter/TrueMemory/blob/main/benchmarks/locomo/EVAL_CONFIG.md)**: exact models, prompts, parameters
220220
221221
---
222222
@@ -237,4 +237,4 @@ Every benchmark script is self-contained and runs on [Modal](https://modal.com).
237237
238238
## ⚖️ License
239239
240-
Licensed under [Apache 2.0](LICENSE). Free for personal and commercial use.
240+
Licensed under [Apache 2.0](https://github.com/buildingjoshbetter/TrueMemory/blob/main/LICENSE). Free for personal and commercial use.

assets/charts/category-heatmap.png

-506 KB
Binary file not shown.

benchmarks/locomo/BENCHMARK_RESULTS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Question categories: **Cat 1** = single-hop, **Cat 2** = multi-hop, **Cat 3** =
4242
| Supermemory | 77.7% (219) | 64.5% (207) | 64.6% (62) | 61.7% (519) | 65.4% |
4343
| Mem0 | 78.0% (220) | 37.7% (121) | 74.0% (71) | 63.5% (534) | 61.4% |
4444

45-
TrueMemory tier per-category numbers are sourced from the authoritative 56-grid sweep (`_working/benchmarks_56combo/results/` in the repo's working archive, files: `mini_b1_model2vec_miniml6_hyde_off_result.json` for Edge, `pro_qwen3_256d_gte_reranker_hyde_off_result.json` for Base, `qwen3_256d__gte_reranker_result.json` for Pro).
45+
TrueMemory tier per-category numbers are sourced from the authoritative 56-grid sweep. The raw result JSONs live in a gitignored working archive (not distributed with the package); the aggregate numbers in the table above are the canonical reference, and the Modal bench scripts in `benchmarks/locomo/scripts/` reproduce them end-to-end.
4646

4747
### Legacy v0.3.0 configurations (deprecated in v0.4.0)
4848

benchmarks/locomo/scripts/bench_truememory_base.py

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
#!/usr/bin/env python3
22
"""
3-
LoCoMo Benchmark — TrueMemory Pro Tier (+HyDE, T4 GPU)
4-
======================================================
5-
TrueMemory Pro tier using Qwen3-Embedding-0.6B @ 256d Matryoshka embeddings,
6-
gte-reranker-modernbert-base (149M params) reranker, and HyDE (Hypothetical
7-
Document Embeddings) via OpenRouter LLM. Paper §2.0 target: 91.8%.
3+
LoCoMo Benchmark — TrueMemory Base Tier (default, HyDE off)
4+
===========================================================
5+
TrueMemory Base tier using Qwen3-Embedding-0.6B @ 256d Matryoshka embeddings
6+
and gte-reranker-modernbert-base (149M params) reranker. HyDE is OFF — Base
7+
is the fully-offline default tier, no LLM API calls. Paper §2.0 target: 91.5%.
88
Requires a T4 GPU on Modal.
99
1010
This is a fully self-contained Modal script. No local imports required.
@@ -15,8 +15,8 @@
1515
Usage:
1616
modal secret create openrouter-key OPENROUTER_API_KEY=sk-or-...
1717
18-
modal run --detach bench_truememory_pro.py # Full run (10 convs, 1540 Qs)
19-
modal run --detach bench_truememory_pro.py --smoke # Smoke test (1 conv, 5 Qs)
18+
modal run --detach bench_truememory_base.py # Full run (10 convs, 1540 Qs)
19+
modal run --detach bench_truememory_base.py --smoke # Smoke test (1 conv, 5 Qs)
2020
2121
modal volume get locomo-results / ./results --force
2222
"""
@@ -186,7 +186,7 @@ def fmsg(m): return f"[{m['timestamp']}] {m['speaker']} to {m['recipient']}: {m[
186186

187187
# ── TrueMemory Context Formatting ─────────────────────────────────────────
188188

189-
def _nm_format_ctx(results):
189+
def _tm_format_ctx(results):
190190
"""Format TrueMemory results with metadata — matches v2 scripts exactly."""
191191
parts = []
192192
for r in results:
@@ -231,7 +231,7 @@ def retrieve_truememory_base(conv_data, conv_idx):
231231
for qa in get_qa(conv_data):
232232
sr = engine.search_agentic(qa["question"], limit=100,
233233
use_hyde=False, use_reranker=True)
234-
ctx = _nm_format_ctx(sr)
234+
ctx = _tm_format_ctx(sr)
235235
results.append((qa["question"], qa["category"], qa["answer"], ctx or "No results found."))
236236
engine.close()
237237
os.unlink(tmp_db)

benchmarks/locomo/scripts/bench_truememory_edge.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -185,7 +185,7 @@ def fmsg(m): return f"[{m['timestamp']}] {m['speaker']} to {m['recipient']}: {m[
185185

186186
# ── TrueMemory Context Formatting ─────────────────────────────────────────
187187

188-
def _nm_format_ctx(results):
188+
def _tm_format_ctx(results):
189189
"""Format TrueMemory results with metadata — matches v2 scripts exactly."""
190190
parts = []
191191
for r in results:
@@ -229,7 +229,7 @@ def retrieve_truememory_edge(conv_data, conv_idx):
229229
results = []
230230
for qa in get_qa(conv_data):
231231
sr = engine.search_agentic(qa["question"], limit=100, use_hyde=False, use_reranker=True)
232-
ctx = _nm_format_ctx(sr)
232+
ctx = _tm_format_ctx(sr)
233233
results.append((qa["question"], qa["category"], qa["answer"], ctx or "No results found."))
234234
engine.close()
235235
os.unlink(tmp_db)

benchmarks/locomo/scripts/bench_truememory_pro.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@ def fmsg(m): return f"[{m['timestamp']}] {m['speaker']} to {m['recipient']}: {m[
186186

187187
# ── TrueMemory Context Formatting ─────────────────────────────────────────
188188

189-
def _nm_format_ctx(results):
189+
def _tm_format_ctx(results):
190190
"""Format TrueMemory results with metadata — matches v2 scripts exactly."""
191191
parts = []
192192
for r in results:
@@ -202,7 +202,7 @@ def _nm_format_ctx(results):
202202
parts.append(f"{meta} {r['content']}")
203203
return "\n\n".join(parts)
204204

205-
def _nm_make_hyde_fn():
205+
def _tm_make_hyde_fn():
206206
"""Create HyDE LLM callable via OpenRouter — matches v2 scripts."""
207207
client = mkc()
208208
def _call(prompt):
@@ -221,7 +221,7 @@ def retrieve_truememory_pro(conv_data, conv_idx):
221221
from truememory.reranker import get_reranker
222222
import tempfile
223223
get_reranker(model_name="Alibaba-NLP/gte-reranker-modernbert-base")
224-
llm_fn = _nm_make_hyde_fn()
224+
llm_fn = _tm_make_hyde_fn()
225225
msgs = parse_conv(conv_data)
226226
_tmp_db_file = tempfile.NamedTemporaryFile(suffix=".db", prefix=f"pro_{conv_idx}_", delete=False)
227227
tmp_db = _tmp_db_file.name
@@ -242,7 +242,7 @@ def retrieve_truememory_pro(conv_data, conv_idx):
242242
for qa in get_qa(conv_data):
243243
sr = engine.search_agentic(qa["question"], limit=100, llm_fn=llm_fn,
244244
use_hyde=True, use_reranker=True)
245-
ctx = _nm_format_ctx(sr)
245+
ctx = _tm_format_ctx(sr)
246246
results.append((qa["question"], qa["category"], qa["answer"], ctx or "No results found."))
247247
engine.close()
248248
os.unlink(tmp_db)

benchmarks/locomo/scripts/modal_benchmark.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -377,7 +377,7 @@ def retrieve_supermemory(conv_data, conv_idx):
377377
results.append((qa["question"], qa["category"], qa["answer"], ctx or "No memories found."))
378378
return results
379379

380-
def _nm_format_ctx(results):
380+
def _tm_format_ctx(results):
381381
"""Format TrueMemory results with metadata — matches v2 scripts exactly."""
382382
parts = []
383383
for r in results:
@@ -393,7 +393,7 @@ def _nm_format_ctx(results):
393393
parts.append(f"{meta} {r['content']}")
394394
return "\n\n".join(parts)
395395

396-
def _nm_make_hyde_fn():
396+
def _tm_make_hyde_fn():
397397
"""Create HyDE LLM callable via OpenRouter — matches v2 scripts."""
398398
client = mkc()
399399
def _call(prompt):
@@ -429,7 +429,7 @@ def retrieve_truememory_base(conv_data, conv_idx):
429429
results = []
430430
for qa in get_qa(conv_data):
431431
sr = engine.search_agentic(qa["question"], limit=100, use_hyde=False, use_reranker=True)
432-
ctx = _nm_format_ctx(sr)
432+
ctx = _tm_format_ctx(sr)
433433
results.append((qa["question"], qa["category"], qa["answer"], ctx or "No results found."))
434434
engine.close()
435435
import os
@@ -447,7 +447,7 @@ def retrieve_truememory_pro(conv_data, conv_idx):
447447
from truememory.reranker import get_reranker
448448
import tempfile
449449
get_reranker(model_name="Alibaba-NLP/gte-reranker-modernbert-base")
450-
llm_fn = _nm_make_hyde_fn()
450+
llm_fn = _tm_make_hyde_fn()
451451
msgs = parse_conv(conv_data)
452452
_tmp_db_file = tempfile.NamedTemporaryFile(suffix=".db", prefix=f"pro_{conv_idx}_", delete=False)
453453
tmp_db = _tmp_db_file.name
@@ -468,7 +468,7 @@ def retrieve_truememory_pro(conv_data, conv_idx):
468468
for qa in get_qa(conv_data):
469469
sr = engine.search_agentic(qa["question"], limit=100, llm_fn=llm_fn,
470470
use_hyde=True, use_reranker=True)
471-
ctx = _nm_format_ctx(sr)
471+
ctx = _tm_format_ctx(sr)
472472
results.append((qa["question"], qa["category"], qa["answer"], ctx or "No results found."))
473473
engine.close()
474474
import os

tests/test_cli_help.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,23 @@ def test_help_long_flag_exits_cleanly():
4545
assert "--version" in result.stdout
4646

4747

48+
def test_help_beats_setup_when_both_passed():
49+
"""Hunter F40 regression lock: `truememory-mcp --setup --help` must
50+
print help and exit 0 WITHOUT running `_setup_claude`. This is the
51+
conventional Unix precedence — docs-emitting flags short-circuit
52+
side-effecting operations. Flipping this would be a CHANGELOG-worthy
53+
behaviour change."""
54+
result = _run_cli(["--setup", "--help"])
55+
assert result.returncode == 0
56+
assert result.stdout.lstrip().startswith("Usage: truememory-mcp"), (
57+
f"expected help banner at start; got:\n{result.stdout}"
58+
)
59+
# `_setup_claude` prints distinctive markers not present in _HELP_TEXT.
60+
assert "Manual setup:" not in result.stdout
61+
assert "existing config preserved" not in result.stdout
62+
assert "stale entry replaced" not in result.stdout
63+
64+
4865
def test_help_short_flag_exits_cleanly():
4966
"""`truememory-mcp -h` must behave identically to --help."""
5067
result = _run_cli(["-h"])

truememory/mcp_server.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1115,6 +1115,12 @@ def main():
11151115

11161116
# Informational flags — these must return immediately, before mcp.run()
11171117
# blocks on stdin or _preload_models() starts background threads.
1118+
#
1119+
# Hunter F40: if both `--help` and `--setup` are passed, `--help` wins
1120+
# (checked first, below). This is the conventional Unix posture —
1121+
# docs-emitting flags short-circuit any side-effecting operation — and
1122+
# is locked in by tests/test_cli_help.py. Flipping the precedence would
1123+
# be a behaviour change worth a CHANGELOG entry.
11181124
if "--help" in argv or "-h" in argv:
11191125
print(_HELP_TEXT)
11201126
return 0

truememory/vector_search.py

Lines changed: 19 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,36 @@
11
"""
22
TrueMemory Vector Search
3-
======================
3+
========================
44
5-
Semantic search using Model2Vec (potion-base-8M, 256-dim) embeddings stored
6-
in a sqlite-vec virtual table. Provides nearest-neighbor retrieval based on
7-
cosine distance so that queries like "networking problems" can surface results
8-
about ECONNREFUSED even when no keywords overlap.
5+
Tier-aware semantic search backed by a sqlite-vec virtual table. The
6+
embedding model is resolved from the active tier when :func:`get_model`
7+
is first called — the module itself imports cleanly with only cheap
8+
constants computed:
9+
10+
edge → Model2Vec potion-base-8M @ 256d (CPU, ~30MB)
11+
base → Qwen3-Embedding-0.6B @ 256d Matryoshka (GPU recommended, ~1.5GB)
12+
pro → Qwen3-Embedding-0.6B @ 256d Matryoshka (GPU recommended, ~1.5GB)
13+
14+
The active tier comes from the ``TRUEMEMORY_EMBED_MODEL`` env var
15+
(``edge`` / ``base`` / ``pro``) or ``~/.truememory/config.json``; MCP-server
16+
callers may also invoke :func:`set_embedding_model` at runtime. Cosine-
17+
distance nearest-neighbour search surfaces queries like "networking
18+
problems" against stored "ECONNREFUSED" messages without keyword overlap.
919
1020
Usage::
1121
12-
from truememory.storage import create_db, load_messages_from_file
22+
from truememory.storage import create_db
1323
from truememory.vector_search import init_vec_table, build_vectors, search_vector
1424
1525
conn = create_db("truememory.db")
16-
load_messages_from_file(conn, "synthetic_v2_messages.json")
26+
# ... insert messages ...
1727
init_vec_table(conn)
1828
build_vectors(conn)
1929
results = search_vector(conn, "networking problems", limit=5)
2030
2131
Dependencies:
22-
- model2vec (``pip install model2vec``)
32+
- model2vec (``pip install model2vec``) — for the edge tier
33+
- sentence-transformers (``pip install truememory[gpu]``) — for base / pro
2334
- sqlite-vec (``pip install sqlite-vec``)
2435
- numpy
2536
"""

0 commit comments

Comments
 (0)