[P2] Memory tier validation script (ADR-017 enforcement)

[rjmurillo-bot]

Context

PR #908 changed 53 memory files, many violating ADR-017 tiered index hierarchy. Need automated enforcement.

Problem Statement

Memory changes violate ADR-017 without detection:

• Orphaned memories outside memory-index.md
• Incorrect tier placement
• Broken cross-references
• No validation in pre-commit or CI

Proposed Solution

Create .agents/scripts/Validate-MemoryTier.ps1:

Checks:

1. All memories referenced in memory-index.md (no orphans)
2. Domain indexes follow {domain}-index.md naming
3. Cross-references valid (files exist)
4. Tier hierarchy respected (index → domain → atomic)

Integration:

• Run in pre-commit hook
• Run in CI (fail on violation)
• Provide clear remediation guidance

Acceptance Criteria

• Script created with all 4 checks
• Exit code 0 = pass, non-zero = fail
• Clear error messages with file paths
• Pre-commit hook integration
• CI workflow integration
• Documentation updated

Effort Estimate

8 hours

Priority

P2 - Normal (This Quarter)

Related

• ADR-017 (tiered memory index architecture)
• PR feat(skill): add reflect skill and auto-learning hook #908 (53 memory files changed)
• Root Cause: Scope Creep via Tool Side Effects

References

.agents/retrospective/2026-01-15-pr-908-comprehensive-retrospective.md (lines 1314-1320)

[traycerai]

Plan

Observations

scripts/validation/memory_index.py already implements all four ADR-017 checks (file references, domain naming, cross-references, tier hierarchy) with a full test suite in tests/test_validation_memory_index.py. The .githooks/pre-commit hook already calls this Python script at lines 1429–1455 (blocking). However, no CI workflow runs it — memory-validation.yml and memory-health.yml only check citation health via memory_enhancement, not ADR-017 tier structure. The .agents/scripts/ directory does not exist.

Approach

ADR-042 mandates Python for new scripts; the logic already lives in scripts/validation/memory_index.py. The ticket explicitly names .agents/scripts/Validate-MemoryTier.ps1, so a thin PowerShell wrapper (per ADR-006: orchestration only, logic in modules) is created to satisfy that requirement without duplicating logic. The primary gap — CI enforcement — is filled with a new workflow following the established memory-validation.yml / memory-health.yml patterns.

---

Implementation Steps

1. Create .agents/scripts/Validate-MemoryTier.ps1

Create the .agents/scripts/ directory and the wrapper script. It must:

• Declare parameters: -MemoryPath (string, default ".serena/memories"), -CI (switch), -Format (string, choices console/markdown/json, default console)
• Verify python3 (or python) is available; exit with code 2 (ADR-035 config error) if not found, printing a clear message: "Python 3 is required. Install Python 3.10+ to run memory tier validation."
• Resolve the path to scripts/validation/memory_index.py relative to the script's own location ($PSScriptRoot/../../scripts/validation/memory_index.py); exit 2 if not found
• Build the argument list: --path $MemoryPath, --format $Format, and --ci when the -CI switch is set
• Invoke the Python script and capture its exit code via $LASTEXITCODE
• On non-zero exit, print remediation guidance:
  • Exit 1: "ADR-017 violations found. Fix issues above, then re-run."
  • Exit 2: "Configuration error. Verify .serena/memories/ exists."
• Return the Python script's exit code via exit $LASTEXITCODE

This keeps all validation logic in the Python module (ADR-006, ADR-042) while providing the PowerShell entry point the ticket requires.

---

2. Create .github/workflows/validate-memory-tier.yml

Follow the exact structural pattern of memory-validation.yml (dorny/paths-filter, skip job, ARM runner, SHA-pinned actions). The workflow must:

Trigger: pull_request (all events) + workflow_dispatch

check-paths job (same pattern as memory-validation.yml):

• runs-on: ubuntu-24.04-arm
• Checkout with the same SHA-pinned actions/checkout
• dorny/paths-filter watching:

  memories:
    - '.serena/memories/**'
  script:
    - 'scripts/validation/memory_index.py'
    - '.github/workflows/validate-memory-tier.yml'

• Output: should-run (true if either filter matches or workflow_dispatch)

validate-memory-tier job (runs when should-run == 'true'):

• runs-on: ubuntu-24.04-arm, timeout-minutes: 10
• Checkout → Setup Python 3.12 → Install UV → uv pip install --system -e ".[dev]"
• Run validation (markdown output for PR comment):

  python3 scripts/validation/memory_index.py \
    --path .serena/memories \
    --ci \
    --format markdown > memory-tier-report.md
  

  Capture exit code in $GITHUB_OUTPUT as validation_exit_code
• Upload memory-tier-report.md as artifact (7-day retention)
• Post PR comment (using actions/github-script, same upsert pattern as memory-validation.yml) with marker <!-- MEMORY-TIER-VALIDATION -->, showing pass/fail status and the markdown report
• Fail the job when exit code is non-zero: exit ${{ steps.validate.outputs.validation_exit_code }} — this makes it a hard gate unlike the existing citation-health workflows

skip-validation job (runs when should-run != 'true'):

• Same pattern as existing skip jobs; prints "No memory changes — skipping ADR-017 tier validation"

All uses: lines must be SHA-pinned with version comments (same SHAs already used in memory-validation.yml).

---

3. Update file:.agents/architecture/ADR-017-tiered-memory-index-architecture.md

In the Confirmation → Validation Implementation Status table, update the script column references:

Validation	Script	Status
File references exist	scripts/validation/memory_index.py:check_file_references	✅
Keyword density ≥40%	scripts/validation/memory_index.py:check_keyword_density	✅
Memory-index references	scripts/validation/memory_index.py:check_memory_index_references	✅
Orphan detection	scripts/validation/memory_index.py:find_orphaned_files	✅
Index entry naming (skill-)	scripts/validation/memory_index.py:check_file_references	✅
Pure lookup table format	scripts/validation/memory_index.py:check_index_format	✅

Add a new Enforcement subsection under Confirmation:

| Gate | Script | Trigger |
|------|--------|---------|
| Pre-commit (blocking) | scripts/validation/memory_index.py | .serena/memories/** staged |
| CI (blocking) | .github/workflows/validate-memory-tier.yml | PR with memory changes |
| Manual / PowerShell | .agents/scripts/Validate-MemoryTier.ps1 | On-demand |

Remove the stale references to Validate-MemoryIndex.ps1 throughout the document.

---

4. Update file:.agents/AGENTS.md

Add a Memory Tier Validator entry to the Automated Actors section (alongside Session Protocol Validator and Consistency Validator):

Attribute	Value
Script	scripts/validation/memory_index.py
PS1 Wrapper	.agents/scripts/Validate-MemoryTier.ps1
Trigger	CI on .serena/memories/** changes; pre-commit hook; manual
Input	.serena/memories/ directory
Output	Console / Markdown / JSON report; exit 0 = pass, 1 = fail

List the four checks it enforces (matching the ticket's acceptance criteria):

1. All memories referenced in memory-index.md (no orphans)
2. Domain indexes follow skills-{domain}-index.md naming
3. Cross-references valid (all referenced files exist)
4. Tier hierarchy respected (pure lookup table format, keyword density ≥40%)

---

Flow Summary

sequenceDiagram
    participant Dev as Developer
    participant Hook as pre-commit hook
    participant PS1 as Validate-MemoryTier.ps1
    participant Py as memory_index.py
    participant CI as validate-memory-tier.yml

    Dev->>Hook: git commit (memory files staged)
    Hook->>Py: python3 scripts/validation/memory_index.py --ci
    Py-->>Hook: exit 0 (pass) or exit 1 (fail)
    Hook-->>Dev: Block commit on failure

    Dev->>PS1: pwsh Validate-MemoryTier.ps1 -CI (manual)
    PS1->>Py: python3 scripts/validation/memory_index.py --ci
    Py-->>PS1: exit code + report
    PS1-->>Dev: Remediation guidance on failure

    Dev->>CI: git push → PR opened
    CI->>Py: python3 scripts/validation/memory_index.py --ci --format markdown
    Py-->>CI: exit code + markdown report
    CI-->>Dev: PR comment + blocking status check

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

`scripts/validation/memory_index.py` already implements all four ADR-017 checks (file references, domain naming, cross-references, tier hierarchy) with a full test suite in `tests/test_validation_memory_index.py`. The `.githooks/pre-commit` hook already calls this Python script at lines 1429–1455 (blocking). However, **no CI workflow runs it** — `memory-validation.yml` and `memory-health.yml` only check citation health via `memory_enhancement`, not ADR-017 tier structure. The `.agents/scripts/` directory does not exist.

## Approach

ADR-042 mandates Python for new scripts; the logic already lives in `scripts/validation/memory_index.py`. The ticket explicitly names `.agents/scripts/Validate-MemoryTier.ps1`, so a thin PowerShell wrapper (per ADR-006: orchestration only, logic in modules) is created to satisfy that requirement without duplicating logic. The primary gap — CI enforcement — is filled with a new workflow following the established `memory-validation.yml` / `memory-health.yml` patterns.

---

## Implementation Steps

### 1. Create `.agents/scripts/Validate-MemoryTier.ps1`

Create the `.agents/scripts/` directory and the wrapper script. It must:

- Declare parameters: `-MemoryPath` (string, default `".serena/memories"`), `-CI` (switch), `-Format` (string, choices `console`/`markdown`/`json`, default `console`)
- Verify `python3` (or `python`) is available; exit with code `2` (ADR-035 config error) if not found, printing a clear message: `"Python 3 is required. Install Python 3.10+ to run memory tier validation."`
- Resolve the path to `scripts/validation/memory_index.py` relative to the script's own location (`$PSScriptRoot/../../scripts/validation/memory_index.py`); exit `2` if not found
- Build the argument list: `--path $MemoryPath`, `--format $Format`, and `--ci` when the `-CI` switch is set
- Invoke the Python script and capture its exit code via `$LASTEXITCODE`
- On non-zero exit, print remediation guidance:
  - Exit 1: `"ADR-017 violations found. Fix issues above, then re-run."`
  - Exit 2: `"Configuration error. Verify .serena/memories/ exists."`
- Return the Python script's exit code via `exit $LASTEXITCODE`

This keeps all validation logic in the Python module (ADR-006, ADR-042) while providing the PowerShell entry point the ticket requires.

---

### 2. Create `.github/workflows/validate-memory-tier.yml`

Follow the exact structural pattern of `memory-validation.yml` (dorny/paths-filter, skip job, ARM runner, SHA-pinned actions). The workflow must:

**Trigger**: `pull_request` (all events) + `workflow_dispatch`

**`check-paths` job** (same pattern as `memory-validation.yml`):
- `runs-on: ubuntu-24.04-arm`
- Checkout with the same SHA-pinned `actions/checkout`
- `dorny/paths-filter` watching:
  ```yaml
  memories:
    - '.serena/memories/**'
  script:
    - 'scripts/validation/memory_index.py'
    - '.github/workflows/validate-memory-tier.yml'
  ```
- Output: `should-run` (true if either filter matches or `workflow_dispatch`)

**`validate-memory-tier` job** (runs when `should-run == 'true'`):
- `runs-on: ubuntu-24.04-arm`, `timeout-minutes: 10`
- Checkout → Setup Python 3.12 → Install UV → `uv pip install --system -e ".[dev]"`
- Run validation (markdown output for PR comment):
  ```
  python3 scripts/validation/memory_index.py \
    --path .serena/memories \
    --ci \
    --format markdown > memory-tier-report.md
  ```
  Capture exit code in `$GITHUB_OUTPUT` as `validation_exit_code`
- Upload `memory-tier-report.md` as artifact (7-day retention)
- Post PR comment (using `actions/github-script`, same upsert pattern as `memory-validation.yml`) with marker `<!-- MEMORY-TIER-VALIDATION -->`, showing pass/fail status and the markdown report
- **Fail the job** when exit code is non-zero: `exit ${{ steps.validate.outputs.validation_exit_code }}` — this makes it a hard gate unlike the existing citation-health workflows

**`skip-validation` job** (runs when `should-run != 'true'`):
- Same pattern as existing skip jobs; prints "No memory changes — skipping ADR-017 tier validation"

All `uses:` lines must be SHA-pinned with version comments (same SHAs already used in `memory-validation.yml`).

---

### 3. Update `file:.agents/architecture/ADR-017-tiered-memory-index-architecture.md`

In the **Confirmation → Validation Implementation Status** table, update the script column references:

| Validation | Script | Status |
|---|---|---|
| File references exist | `scripts/validation/memory_index.py:check_file_references` | ✅ |
| Keyword density ≥40% | `scripts/validation/memory_index.py:check_keyword_density` | ✅ |
| Memory-index references | `scripts/validation/memory_index.py:check_memory_index_references` | ✅ |
| Orphan detection | `scripts/validation/memory_index.py:find_orphaned_files` | ✅ |
| Index entry naming (skill-) | `scripts/validation/memory_index.py:check_file_references` | ✅ |
| Pure lookup table format | `scripts/validation/memory_index.py:check_index_format` | ✅ |

Add a new **Enforcement** subsection under Confirmation:

```
| Gate | Script | Trigger |
|------|--------|---------|
| Pre-commit (blocking) | scripts/validation/memory_index.py | .serena/memories/** staged |
| CI (blocking) | .github/workflows/validate-memory-tier.yml | PR with memory changes |
| Manual / PowerShell | .agents/scripts/Validate-MemoryTier.ps1 | On-demand |
```

Remove the stale references to `Validate-MemoryIndex.ps1` throughout the document.

---

### 4. Update `file:.agents/AGENTS.md`

Add a **Memory Tier Validator** entry to the Automated Actors section (alongside Session Protocol Validator and Consistency Validator):

| Attribute | Value |
|---|---|
| **Script** | `scripts/validation/memory_index.py` |
| **PS1 Wrapper** | `.agents/scripts/Validate-MemoryTier.ps1` |
| **Trigger** | CI on `.serena/memories/**` changes; pre-commit hook; manual |
| **Input** | `.serena/memories/` directory |
| **Output** | Console / Markdown / JSON report; exit 0 = pass, 1 = fail |

List the four checks it enforces (matching the ticket's acceptance criteria):
1. All memories referenced in `memory-index.md` (no orphans)
2. Domain indexes follow `skills-{domain}-index.md` naming
3. Cross-references valid (all referenced files exist)
4. Tier hierarchy respected (pure lookup table format, keyword density ≥40%)

---

### Flow Summary

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant Hook as pre-commit hook
    participant PS1 as Validate-MemoryTier.ps1
    participant Py as memory_index.py
    participant CI as validate-memory-tier.yml

    Dev->>Hook: git commit (memory files staged)
    Hook->>Py: python3 scripts/validation/memory_index.py --ci
    Py-->>Hook: exit 0 (pass) or exit 1 (fail)
    Hook-->>Dev: Block commit on failure

    Dev->>PS1: pwsh Validate-MemoryTier.ps1 -CI (manual)
    PS1->>Py: python3 scripts/validation/memory_index.py --ci
    Py-->>PS1: exit code + report
    PS1-->>Dev: Remediation guidance on failure

    Dev->>CI: git push → PR opened
    CI->>Py: python3 scripts/validation/memory_index.py --ci --format markdown
    Py-->>CI: exit code + markdown report
    CI-->>Dev: PR comment + blocking status check
```

---

Execution Information

Branch: main
Commit: 94e7a50

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#301 - docs: autonomous PR monitoring prompt and retrospective [merged]
#332 - docs: add feature request review workflow planning artifacts [merged]
#342 - ci: add memory validation workflow for ADR-017 enforcement [merged]
#365 - fix(memory): ADR-017 compliance - rename skill- files and implement P0 validations [merged]
#730 - feat: ADR-007 Memory-First Architecture enforcement and Forgetful MCP setup [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[github-actions]

Product Requirements Document

Important

This PRD was automatically generated based on issue triage escalation.
Complexity Score: 7/12 (Detailed)
Escalation Criteria: feature_request,multi_phase_work

About this PRD

This PRD was generated by the Explainer Agent after the issue was flagged
for detailed analysis. The document is designed to be self-contained - it can
be passed to a separate AI agent instance for implementation without additional
context.

PRD Depth Levels:

• Standard (4-6): Brief analysis, clear requirements
• Detailed (7-9): Research section, blocking questions
• Comprehensive (10-12): Phased implementation, risk analysis

---

I now have sufficient context. Let me check if there's a retrospective file mentioned in the issue for additional context.

The file system is read-only. I'll output the PRD content directly for you to save.

---

PRD: Memory Tier Validation Script (ADR-017 Enforcement)

Executive Summary

PR #908 modified 53 memory files with violations of ADR-017's tiered index hierarchy undetected until post-merge review. The existing Validate-MemoryIndex.ps1 validates domain index structure but lacks validation for complete memory tier hierarchy including orphan detection across all tiers and cross-reference validity.

Scope	Status	Blocker
All memories referenced in memory-index.md	🟢 READY	None
Domain index naming convention	🟢 READY	None
Cross-reference validation	🟢 READY	None
Tier hierarchy enforcement	🟢 READY	None
Pre-commit hook integration	🟢 READY	None
CI workflow integration	🟢 READY	None

Verdict: READY

---

Problem Statement

Current State

Component	Status	Gap
Validate-MemoryIndex.ps1	Exists	Validates domain indices only, not full hierarchy
memory-validation.yml	Exists	Runs Validate-MemoryIndex.ps1, Validate-SkillFormat.ps1
Pre-commit hook	Exists	No memory validation integrated
memory-index.md	Exists	Top-level index with keyword-to-file mappings

Gap Analysis

Current Validate-MemoryIndex.ps1 focuses on domain index validation but does not:

1. Validate memory-index.md completeness
2. Detect orphaned atomic memories not referenced by any index
3. Validate cross-references within atomic files (## Related sections)
4. Enforce tier hierarchy (Level 0 → Level 1 → Level 2)

Evidence: PR #908 changed 53 memory files. Post-merge analysis revealed orphaned memories, incorrect tier placement, and broken cross-references.

---

Research Findings

ADR-017 Confirmation: Mandates pre-commit and CI validation but current implementation is incomplete.

Existing Functions in Validate-MemoryIndex.ps1:

• Test-FileReferences - Implemented
• Test-KeywordDensity - Implemented
• Test-IndexFormat - Implemented
• Test-MemoryIndexReferences - Implemented
• Get-OrphanedFiles - Implemented (domain-prefixed only)

Gap: Orphan detection only checks domain-prefixed files. Does not detect atomic files with non-standard names or validate cross-references.

---

Proposed Solution

Phase 1: Extend Validate-MemoryIndex.ps1 (4 hours)

Task	Description
1.1	Add Test-AtomicFileOrphans function
1.2	Add Test-CrossReferences function
1.3	Add Test-TierHierarchy function
1.4	Add -TierValidation switch
1.5	Add clear remediation guidance

Phase 2: Pre-commit Hook Integration (2 hours)

Task	Description
2.1	Add memory validation section to .githooks/pre-commit
2.2	Run on staged .serena/memories/ files
2.3	Use warning mode (non-blocking) initially
2.4	Document SKIP_MEMORY_VALIDATION bypass

Phase 3: CI Integration Enhancement (1 hour)

Task	Description
3.1	Update memory-validation.yml with -TierValidation flag
3.2	Exit code 1 on tier violations (blocking)

Phase 4: Documentation (1 hour)

Task	Description
4.1	Update ADR-017 validation status table
4.2	Add usage examples to script help

---

Functional Requirements

FR-1: Orphan Detection (P0)

ID	Requirement	Acceptance Criteria
FR-1.1	Detect atomic files not referenced by any domain index	Script lists all unindexed .md files
FR-1.2	Detect domain indices not referenced by memory-index.md	Script lists orphaned indices
FR-1.3	Exclude known non-indexed files	Skip memory-index.md, subdirectories
FR-1.4	Provide remediation guidance	Error includes which index should reference the orphan

FR-2: Cross-Reference Validation (P0)

ID	Requirement	Acceptance Criteria
FR-2.1	Validate ## Related section links	All links resolve to existing files
FR-2.2	Report broken references with location	Error includes source file, line number, target

FR-3: Tier Hierarchy Validation (P1)

ID	Requirement	Acceptance Criteria
FR-3.1	Validate Level 0 → Level 1	Every domain index in memory-index.md exists
FR-3.2	Validate Level 1 → Level 2	Every atomic file in domain indices exists

FR-4: Exit Codes (P0)

ID	Requirement	Acceptance Criteria
FR-4.1	Exit 0 on success	No violations returns 0
FR-4.2	Exit 1 on P0 violations	Orphans or broken references return 1

---

Technical Design

Script Extension

param(
    [switch]$TierValidation  # Enable full hierarchy check
)

function Test-AtomicFileOrphans { ... }
function Test-CrossReferences { ... }
function Test-TierHierarchy { ... }

Pre-commit Hook Addition

STAGED_MEMORY_FILES=$(echo "$STAGED_FILES" | grep -E '^\.serena/memories/.*\.md$' || true)

if [ -n "$STAGED_MEMORY_FILES" ]; then
    pwsh -NoProfile -File "$REPO_ROOT/scripts/Validate-MemoryIndex.ps1" -TierValidation
fi

CI Workflow Enhancement

- name: Validate Memory Tier Hierarchy
  run: ./scripts/Validate-MemoryIndex.ps1 -CI -TierValidation

---

Risks and Mitigations

Risk	Likelihood	Impact	Mitigation
False positives block PRs	Medium	High	Non-blocking initially; tune before enforcement
Performance with 500+ files	Low	Medium	Lazy loading; only scan changed files
Breaking existing workflow	Low	High	Add -TierValidation flag; don't change defaults

---

Success Metrics

Metric	Target
Orphan detection accuracy	100%
Cross-reference validation	100% of ## Related sections
Pre-commit execution	<5 seconds
Test coverage	>=80% for new functions

---

Implementation Status

Validation	Status
File references exist	Implemented
Keyword density	Implemented
Memory-index references	Implemented
Domain orphan detection	Implemented
Atomic file orphans	TO IMPLEMENT
Cross-references	TO IMPLEMENT
Tier hierarchy	TO IMPLEMENT

Total Effort: 8 hours | Priority: P2

_{Generated by AI PRD Generation | View Workflow}

[github-actions]

AI Triage Summary

Note

This issue has been automatically triaged by AI agents

What is AI Triage?

This issue was analyzed by AI agents:

• Analyst Agent: Categorizes the issue and suggests appropriate labels
• Roadmap Agent: Aligns the issue with project milestones and priorities
• Explainer Agent (if escalated): Generates comprehensive PRD

Triage Results

Property	Value
Category	enhancement
Labels	["enhancement","agent-memory","area-infrastructure"]
Priority	P2
Milestone	v1.1
PRD Escalation	Generated (see below)

Categorization Analysis

```json
{
  "labels": ["enhancement", "agent-memory", "area-infrastructure"],
  "category": "enhancement",
  "confidence": 0.9,
  "reasoning": "Issue requests a new validation script (Validate-MemoryTier.ps1) to enforce ADR-017 memory tier hierarchy with CI/pre-commit integration"
}

</details>

<details>
<summary>Roadmap Alignment</summary>

```json
```json
{
  "milestone": "v1.1",
  "priority": "P2",
  "epic_alignment": "",
  "confidence": 0.70,
  "reasoning": "Tooling enhancement for ADR enforcement fits v1.1 maintainability focus but does not align to existing epics.",
  "escalate_to_prd": true,
  "escalation_criteria": ["feature_request", "multi_phase_work"],
  "complexity_score": 7
}

</details>

---

<sub>Powered by [AI Issue Triage](https://github.com/rjmurillo/ai-agents) workflow</sub>