Memory System Audit: 28 Writers, 2 Readers, 9 Gaps — What We Found and Fixed

[jlacour-git]

Memory System Audit: Producer/Consumer Gaps and Fixes

TL;DR

We built a custom session summary system (writer + retrieval) and modified WorkCleanup to manage the session lifecycle. In doing so, we introduced two integration mistakes that went undetected until a cross-session handoff failed. Investigating those mistakes triggered a full producer/consumer audit of the MEMORY/ directory. The audit uncovered 28 writers, 2 readers, and 9 gaps where data was written but never retrieved. The most critical: our own session summaries were being written but never read back, and WorkCleanup was about to start permanently deleting session data without proper archival. All gaps are now fixed.

This post shares what we got wrong, what we found, and the resulting architecture — so others can avoid the same mistakes and strengthen their own PAI memory systems.

---

What We Got Wrong

We were building custom improvements to the memory system across multiple sessions. In one session, we added a session summary writer. In another, we built the ContextAssembler to retrieve context. Each piece worked on its own — but we never verified the end-to-end data flow. That's how two integration gaps slipped in.

The trigger: we tried to hand off work between sessions. A prior session analyzed Algorithm v3.6.0 (PR #871) and said: "Start a fresh session — I'll pick up the context via session summary." The new session cold-started with zero context.

Our mistake #1: When building getRecentWork() in ContextAssembler, we added a skip for the summaries/ directory (correct — summaries aren't work entries). But we never built the corresponding getRecentSummaries() scanner. The prds/ skip had a dedicated scanner, summaries/ didn't. The data was being written correctly — we just forgot to build the reader.

// This skip was correct — but nobody built the scanner for summaries
if (entry.name === 'summaries' || entry.name === 'prds') continue;

Our mistake #2: We built getArchivedPRDs() to scan MEMORY/WORK/prds/ — but the archival process that was supposed to populate that directory was never implemented. The scanner worked perfectly; it just scanned an empty directory. 40 PRDs sat in their original work directories, becoming invisible once the 7-day recency cutoff expired.

Both mistakes share the same root cause: building producers and consumers in separate sessions without integration testing. This prompted a full system audit.

---

The Audit

Method

We mapped every producer (file writer) and every consumer (file reader) across the entire MEMORY/ directory tree, then identified every gap where written data had no retrieval path.

Scale

• 28 distinct writers produce data into MEMORY/ (hooks, handlers, tools, CLI utilities)
• 2 retrieval points read from MEMORY/:
  • LoadContext.hook.ts — once at session startup
  • ContextAssembler.hook.ts — every prompt via UserPromptSubmit
• Everything else (AlgorithmTracker, PRDSync, tab-setter, etc.) reads/writes for operational state, not context retrieval

Gaps Found

ID	Gap	Severity	Data at Risk
GAP-1	Session summaries (WORK/summaries/) — no reader	CRITICAL	144 summary files invisible to all sessions
GAP-2	WorkCleanup destroys META.yaml, THREAD.md, ISC.json	CRITICAL	All non-PRD session data permanently deleted after 7 days
GAP-3	LEARNING/ALGORITHM + LEARNING/SYSTEM — no per-turn retrieval	MODERATE	302 learning files invisible mid-session
GAP-4	Session summaries not in LoadContext startup	MODERATE	No cross-session context at startup
GAP-5	RESEARCH output (RESEARCH/*.md) — no reader	LOW	Subagent research results never resurfaced
GAP-6	LEARNING/SYNTHESIS — no reader	LOW	Pattern synthesis reports invisible
GAP-7	THREAD.md files (134) — no reader	MODERATE	Task-level context invisible (scaffolding, acceptable)
GAP-8	Orphaned STATE files never cleaned	LOW	context-seen-.json and algorithms/.json accumulate
GAP-9	WISDOM/FRAMES/ and progress/ dirs don't exist	BENIGN	Dead code paths in LoadContext

Growth Assessment

Several directories grow unbounded with no cleanup mechanism:

Directory	Current Files	Growth Rate
LEARNING/ALGORITHM/	228	~3-5/day
LEARNING/SYSTEM/	36	~1/day
WORK/summaries/	144	~5-10/day
SECURITY/	35	~5/day
VOICE/voice-events.jsonl	210 KB	~10KB/day
STATE/context-seen-*	14	~1-3/day

---

The Fixes

Fix 1: ContextAssembler — Coverage Gap Repairs

Added getRecentSummaries() — scans WORK/summaries/*.md with keyword matching + 7-day recency. Type weight: 0.85 (between current-work 1.0 and recent-work 0.8). Cap: 5 results.

Replaced getArchivedPRDs() with getKeywordMatchedPRDs() — scans both WORK/ and ARCHIVE/ (see Fix 2) by keyword with no recency cutoff. Deduplicates against recent work. Cap: 5 results.

Added getRelevantLearnings() — scans LEARNING/ALGORITHM/ and LEARNING/SYSTEM/ by keyword. Type weight: 0.45. Cap: 3 results. This surfaces past learning signals relevant to the current task.

Files changed: PAI/Tools/ContextAssembler.ts, hooks/ContextAssembler.hook.ts

Fix 2: WorkCleanup — Archive Instead of Delete

Complete rewrite. Old behavior: copy PRD.md to prds/, then rmSync(dir, recursive). New behavior:

• Move (not delete) old session dirs to MEMORY/ARCHIVE/ via renameSync
• Session summaries stay in WORK/summaries/ (hot path — always scanned)
• All session content preserved: META.yaml, THREAD.md, ISC.json, PRD.md, scratch/
• Consolidated STATE cleanup: removes orphaned context-seen-*.json for sessions not in session-names.json, removes algorithms/*.json older than 7 days
• Removed empty prds/ directory (archival concept replaced by ARCHIVE/)

Architecture:

MEMORY/
  WORK/
    summaries/          <- hot: always scanned by ContextAssembler
    {active-sessions}/  <- sessions < 7 days
  ARCHIVE/
    {old-sessions}/     <- sessions > 7 days, full content preserved
                           PRDs searchable by keyword via ContextAssembler

File changed: hooks/WorkCleanup.hook.ts

Fix 3: LoadContext — Session Summaries at Startup

Added "Recent Session Summaries" section to the startup banner. Loads 3 most recent summaries with heading + 150-character preview. Gives immediate cross-session context before the first ContextAssembler injection.

File changed: hooks/LoadContext.hook.ts

---

Resulting Memory Architecture

The Two-Temperature Model

Hot (always scanned, budget-managed):

• WORK/summaries/ — session summaries (ContextAssembler + LoadContext)
• WORK/{active-sessions}/ — PRDs, META (ContextAssembler, 7-day window)
• LEARNING/FAILURES/ — failure contexts (ContextAssembler, keyword-matched)
• LEARNING/ALGORITHM/ + LEARNING/SYSTEM/ — learning signals (ContextAssembler, keyword-matched)
• Project memory files — standalone .md files (ContextAssembler, keyword-matched)

Cold (preserved, searchable on demand):

• ARCHIVE/{old-sessions}/ — full session dirs after 7 days (PRDs keyword-searchable)
• LEARNING/SIGNALS/ratings.jsonl — raw signal data (consumed via learning-cache.sh)
• SECURITY/, VOICE/, RESEARCH/ — audit trails and diagnostics

Key Design Principle

Summaries are the card catalog; session dirs are the filing cabinet. Summaries stay hot forever (in WORK/summaries/). Session dirs go cold after 7 days (moved to ARCHIVE/). The summary captures the decisions and outcomes; the session dir preserves the raw materials for forensic recovery if needed.

---

Lessons Learned

1. Producer-consumer imbalance is the systemic pattern. It's easy to add a writer (a new hook that captures data). It's easy to forget the corresponding reader. The result: data accumulates but is invisible. Any memory system should audit its read paths as carefully as its write paths.

2. Integration testing for data flow. These bugs were built in separate sessions. Session A added the summary writer. Session B built the ContextAssembler. Neither verified the end-to-end flow. A simple test — "write a summary, then run ContextAssembler and check if it surfaces" — would have caught both gaps immediately.

3. A skip without a redirect is a silent drop. When we added if (entry.name === 'summaries') continue to getRecentWork(), it was correct — summaries aren't work entries. But we treated the skip as "handled" when it was really "deferred." The corresponding scanner was our responsibility to build, and we didn't. Any time you skip a directory in a scanner, ask: "who reads this instead?"

4. Archive before delete. The original WorkCleanup was designed to be lightweight — just clean up old dirs. But "clean up" meant "permanently destroy." For a memory system, destruction should be the exceptional case, not the default. Move-to-archive costs the same as delete but preserves recoverability.

---

Implementation Guide

If you want to apply these fixes to your own PAI install:

1. ContextAssembler.ts — Add the session-summary and learning types to the Candidate union. Add getRecentSummaries(), getRelevantLearnings(), and update getKeywordMatchedPRDs() to also scan ARCHIVE/. Wire all three into assemble().

2. ContextAssembler.hook.ts — Update the display grouping ternaries to handle the new types.

3. WorkCleanup.hook.ts — Replace rmSync with renameSync to MEMORY/ARCHIVE/. Add STATE file cleanup. Remove the prds/ directory concept.

4. LoadContext.hook.ts — Add a summary section to checkActiveProgress() that reads 3 most recent files from WORK/summaries/.

The detailed audit report with the full 28-writer / 22-reader matrix is available as a gist (linked below).

---

Discovered and fixed on PAI v4.0.3. All changes tracked as local patches (#31, #32, #33).

[KimVatnedal]

Great work! Are the changes available in a gist?

[DolphusCY]

@jlacour-git Hey J. Could you post the gist for this. I want to implement it so I can make sure that the follow up to #828 after two days is more accurate. Another question... thanks

[jlacour-git]

Certainly, gentlemen!

[jlacour-git]

Thanks @KimVatnedal! Here are the four files:

https://gist.github.com/jlacour-git/b3d465e0b8e505420dd5b38958d2364e

• ContextAssembler.ts — the main assembler (keyword extraction, scoring, budget management)
• ContextAssembler.hook.ts — the UserPromptSubmit hook that calls it per-turn with dedup
• WorkCleanup.hook.ts — archive-instead-of-delete for old session dirs
• LoadContext.hook.ts — SessionStart hook that injects session summaries + active work

Note: ContextAssembler.ts imports from hooks/lib/paths.ts for PAI directory resolution. If you're adapting these, you'll need to adjust those imports to match your setup.

[jlacour-git]

Updated the gist with a small improvement to ContextAssembler.hook.ts.

The dedup manifest now tracks token counts per file, so the feedback line shows both total context burden and per-turn delta:

Context assembled: 2 new files (1 summaries, 1 work), 4,971/5,000 tokens [1,013 added, 8 dedup-skipped]

Previously it only showed the delta — which made the context load look trivially small on later turns when most files got deduped. Now you see at a glance what the model is actually carrying.

[KimVatnedal]

Great 😀 I see there are more fixes in other discussions, do you recommend implementing them all while we wait for official releases? Sent from [Proton Mail](https://proton.me/mail/home) for Android.

…

-------- Original Message --------

On Wednesday, 03/04/26 at 12:40 jlacour-git ***@***.***> wrote: Updated the gist with a small improvement to ContextAssembler.hook.ts. The dedup manifest now tracks token counts per file, so the feedback line shows both total context burden and per-turn delta: Context assembled: 2 new files (1 summaries, 1 work), 4,971/5,000 tokens [1,013 added, 8 dedup-skipped] Previously it only showed the delta — which made the context load look trivially small on later turns when most files got deduped. Now you see at a glance what the model is actually carrying. — Reply to this email directly, [view it on GitHub](#884 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ANZDKXSU7HLY4Q65FRT2VD34PAI27AVCNFSM6AAAAACWFI2KWCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKOJZGY4DENY). You are receiving this because you were mentioned.Message ID: ***@***.***>

[jlacour-git]

Updated the gist again — now 5 files:

https://gist.github.com/jlacour-git/b3d465e0b8e505420dd5b38958d2364e

What changed:

The ContextAssembler.hook.ts got a proper retrieval gate. The original version injected context on every single turn, even "yes" or "ok fine." Over a 20-turn session, that added up to almost 2x the 5K budget — lots of noise, especially from low-signal keyword matches like "fine" appearing in learning files.

The fix is a 3-tier gate:

1. Heuristic — acknowledgments and very short messages skip instantly (zero cost)
2. Haiku extraction — ambiguous prompts get sent to Haiku, which either says SKIP or returns targeted search terms. Much better than keyword extraction on short chat messages
3. Normal retrieval — rich prompts go through as before

Also added self-exclusion: files created during the current session don't get re-injected back to you.

New file: CorrectionMode.hook.ts — a cascade breaker. When you prefix a prompt with stop: followed by feedback, it forces the model to use external grounding (tools or thinking skills) before responding. Based on research showing self-correction without new evidence degrades performance (Huang et al. 2024). Early experiment, but promising.

Re: implementing fixes from other discussions — I'd be selective. The ones that fix clear bugs (like the surrogate pair splitting in RatingCapture, PR #882) are safe. Architectural changes are riskier without understanding how they interact with your local setup. The LOCAL_PATCHES.md approach helps a lot — track what you changed so upgrades don't silently overwrite your fixes. Happy to share that system too if useful.

[KimVatnedal]

Thanks, would love that. Im keeping a fork, and its getting kinda messy 😂

[jlacour-git]

ContextAssembler Update: Semantic Search Overhaul (2026-03-24)

Gist updated: https://gist.github.com/jla/b3d465e0b8e505420dd5b38958d2364e
Files changed: ContextAssembler.ts, BuildSemanticIndex.ts, semantic-config.ts (new)

After extended production use of our ContextAssembler (shared earlier in this thread), we discovered and fixed three fundamental issues with the semantic search layer.

Problem

1. nomic-embed-text produces degenerate vectors for short text. Chunks under ~50 characters (section headings, template placeholders) produced embeddings that scored 0.978 cosine similarity against completely unrelated queries. 55% of our index was affected.

2. nomic-embed-text is language-blind for non-English. Our bilingual corpus (English + German) had German documents scoring 1.000 against each other — the model couldn't distinguish different documents at all.

3. Additive scoring formula let irrelevant files through. Files with zero semantic relevance could score 0.42 from recency + type weight alone, passing the threshold.

Changes

Model swap: nomic-embed-text → mxbai-embed-large (334M, 1024d, multilingual). Garbage max cosine: 0.978 → 0.476. German discrimination: 1.000 → 0.581. 4/4 clean per-query separation (was 2/4). 6ms slower per embed — negligible.

S-gate scoring architecture. Replaced R*0.2 + S*0.5 + T*0.3 with semantic-relevance-as-gatekeeper: if raw cosine < 0.50, the file is excluded — period. R and T only break ties among relevant files. This means S=0 (no semantic match) → score 0, regardless of recency or document type.

Indexer quality gate: min 50 chars body. Prevents degenerate short chunks from entering the index. Eliminated 55% of indexed chunks that were heading-only or placeholder content.

Results

Metric	Before	After
Index vectors	4,297	2,672
Garbage max score	1.000	0.415
Cross-project bleed	Loaded (score 0.42)	Excluded (score 0.000)
Relevant query candidates	58	30
Irrelevant query candidates	58	12

Key insight

Test with deliberately absurd queries. We used "Kochen Rezept Kartoffeln" (cooking recipe potatoes) against an infrastructure knowledge base. It exposed the degenerate vector problem immediately — something threshold-tuning would have masked.

Config: ollama pull mxbai-embed-large + bun BuildSemanticIndex.ts --full. See semantic-config.ts in the gist for all constants.