Closing the Learning Loop — A Digest Skill for Periodic Signal Review

[jlacour-git]

Following up on our Memory System Audit (#884) — we shared what we found when we audited the learning/memory capture pipeline (28 writers, 2 readers, 9 gaps). One piece we've since built and battle-tested is a DigestLearnings skill that periodically reviews all accumulated learning signals and extracts actionable improvement proposals.

The Problem It Solves

PAI captures learning signals — failure analyses, low-rated interactions, algorithm reflections — into MEMORY/LEARNING/. But without periodic review, those signals just accumulate. The AI doesn't spontaneously go back and ask "what patterns do these failures reveal?" You need a structured process.

How It Works

Three phases: Digest → Classify → Track.

1. Digest — Watermark-based incremental scan. Reads a JSONL log for the last processed timestamp, only reviews new files. Deduplicates (the capture system over-records — same incident often appears as FAILURE + ALGORITHM + SYSTEM signal). Maps each unique incident to existing rules or identifies genuine gaps.

2. Classify — Each proposal gets tagged: USER-SAFE (apply directly), SYSTEM-PATCH (needs LOCAL_PATCHES.md + upstream issue), or UPSTREAM-ONLY (file issue, no local change). Decisions collected via structured questions, not a flat report.

3. Track — Approved changes applied, watermark updated. Full audit trail in DIGEST-LOG.jsonl.

What We Learned After 7 Digests

The first few digests found genuine rule gaps — things like "check before creating files" and "preserve facts during tone edits." But after ~4 runs, the rule set stabilized. Since then, every digest has found 0 new rule proposals. The signals are almost entirely adherence failures of existing rules, not missing rules. This is actually the expected mature state.

The digest skill is now more of a health check than a gap-finder — it confirms the rule set is complete and quantifies the adherence problem.

Full Skill Code

The complete workflow (scan logic, classification scheme, watermark management, and 7-digest results table) is available as a gist:

👉 Gist: DigestLearnings Skill

Connection to Prior Work

• #884 — Memory System Audit — The audit that exposed the "28 writers, 2 readers" gap. The digest skill is one of the readers we built.
• #693 — Making the Learning System Actionable — Our earlier proposal for USER-tier steering rules derived from learning signals. The digest skill automates that process.
• #945 — Rule Adherence — The adherence investigation that confirmed "rules are complete, execution is the problem." The digest's 0-proposal streak was key evidence.

Would be curious if others have built similar periodic review mechanisms, or if you've found different approaches to closing the learning loop.

[virtualian]

@jlacour-git Excellent, I'm working on this too.

Did you see my questions over here?

[Drizzt321]

@jlacour-git So I see you link to a gist, has a single file, doesn't look like it's a complete Skill.

Looking at all of your gists, I see:

gist-digest-learnings-skill.md
README.md
AISTEERINGRULES.md
01-LOCAL_PATCHES-TEMPLATE.md
BuildSemanticIndex.ts
Algorithm-Complexity-Gate.md

I don't see the the files you laid out at the end of your link (https://gist.github.com/jlacour-git/bb9e8b6e88ce7e6afa20fd4251beca37). I'm wanting to try yours out, but failing to identify where to get the actual Skill files that you've describe. Am I blind or missing something obvious?

skills/Digest/
├── SKILL.md              # Skill metadata and trigger words
└── Workflows/
    └── DigestLearnings.md  # Full workflow instructions

[virtualian]

I think I just asked CC to 'apply this [gist-link] to my PAI installation'... your mileage may vary :-)

[Drizzt321]

You mean the gist contains all of the info in all of those files, just not split up into those files?

[virtualian]

Apologies, I see what you mean now.

Full Skill Code

The complete workflow (scan logic, classification scheme, watermark management, and 7-digest results table) is available as a gist:

👉 Gist: DigestLearnings Skill

This description doesn't look accurate; the file looks like a digest report, not the full skill.

@jlacour-git?

[jlacour-git]

Hey @Drizzt321! You're not blind. The gist only had the SKILL.md — the workflow file was missing. Just added it.

Here's the complete structure:

skills/Digest/
├── SKILL.md              # Skill metadata and trigger words
└── Workflows/
    └── DigestLearnings.md  # Full workflow instructions

Both files are now in the gist: https://gist.github.com/jlacour-git/bb9e8b6e88ce7e6afa20fd4251beca37

Quick orientation:

• Phase 1 scans MEMORY/LEARNING/ for unprocessed signals, uses a watermark file to avoid re-reading
• Phase 2 classifies each proposal as USER-SAFE / SYSTEM-PATCH / UPSTREAM-ONLY
• Phase 3 applies approved changes and updates the watermark

The key design choice: nothing gets applied without explicit approval. The skill proposes, you decide.

@virtualian — saw your cross-reference from #908. Responding there!