Memory system entity design and refashioning

Major improvements to official memory server usage:
- Discovered search is keyword-based, not semantic - critical for entity design
- Developed entity selection guidelines: broad, stable, searchable entities
- Successfully refashioned unwieldy 100+ observation 'user' entity into 5 focused entities:
  * Socratic Shell project (repository structure, documentation)
  * Memory experimentation (research approaches, philosophy)
  * Blog post development (idea-oriented programming, PL insights)
  * Voice for writing (communication patterns, style guides)
  * Collaborative prompting patterns (workflows, consolidation moments)

Enhanced memory server integration:
- Updated official memory server prompt with comprehensive entity guidelines
- Added 'survey existing entities first' to 'Hi again, Claude' practice
- Updated official memory server README with concise guidance for external users

Key insights:
- User-centric entities create discovery problems - focus on work/concepts instead
- Keyword-based search requires deliberate naming strategies
- Memory system now much more discoverable and useful for collaboration

Memory structure transformed from single catch-all entity to focused,
searchable entities that align with natural work patterns.

Author: nikomatsakis

src/memory-bank/current-state.md

@@ -2,13 +2,27 @@
 
 ## Recent Progress
 
+### Memory System Entity Design (July 2025)
+- **Search functionality testing**: Discovered official memory server uses keyword-based search, not semantic search
+- **Entity design guidelines**: Developed principles for creating broad, stable, searchable entities instead of narrow or user-centric ones
+- **Memory refashioning**: Successfully transformed unwieldy 100+ observation "Niko" entity into 5 focused entities:
+  - "Socratic Shell project" - Repository structure and documentation
+  - "Memory experimentation" - All memory system research and approaches  
+  - "Blog post development" - Writing projects and programming language insights
+  - "Voice for writing" - Communication patterns and style guides
+  - "Collaborative prompting patterns" - Interaction methods and workflows
+- **Updated memory prompt**: Enhanced official memory server prompt with entity selection guidelines
+- **Documentation updates**: Updated official memory server README with concise guidance for external users
+
 ### Documentation Restructuring (July 2025)
 - **Memory approaches organization**: Restructured documentation to organize memory approaches with consistent structure (main README + associated prompts)
 - **Retaining context improvements**: Enhanced introduction with collaborative partnership framing, added "Different audiences" section highlighting individual vs. shared knowledge needs
 - **Navigation cleanup**: Removed redundant "Per-project prompts" section, updated all cross-references to new structure
 - **Voice alignment**: Applied "Niko voice" principles throughout - practical over theoretical, direct about challenges, experience-driven
 
 ### Key Insights Captured
+- **Search limitations matter**: Keyword-based search requires deliberate entity naming and organization strategies
+- **User-centric entities are problematic**: Better to create entities for work/concepts with collaborative insights embedded
 - **Different audiences need**: Recognition that memory systems must serve both individual collaboration history and shared project knowledge
 - **Productive desynchronization**: Individual memory can drift from project memory (e.g., out-of-date rustc knowledge) while still being useful
 - **Organizational systems**: Claude won't remember everything, so we need systems to pull context in on demand without overwhelming

src/official-memory-server/README.md

@@ -18,10 +18,6 @@ Rather than using the memory server mechanically, we've fashioned a [custom prom
 
 This approach treats the external knowledge graph not as a database to fill but as a way to preserve the collaborative understanding that develops between human and AI over time.
 
-## Integration Notes
-Testing how external knowledge graphs support:
-- Organic memory updates during consolidation moments
-- Dual memory layers (shared project + personal user context)
-- Productive desynchronization as a feature rather than bug
+## Entity Design Guidelines
 
-The focus is on storing insights and patterns rather than raw information, using relationships to capture how understanding connects and evolves.
+The official memory server uses keyword-based searching. By default, Claude was simply glomming all the memories onto a single entity that represented the user. The prompt therefore includes guidance meant to improve memory retrieval by focusing memories on a small set of entities, and recognizing that all the memories in the file are always "relative" to the current user anyhow.

src/prompts/project/official-memory-server.md

@@ -8,7 +8,7 @@ Memory in our collaboration is not a database to be filled but a living dimensio
 
 ## When Memory Arises
 
-**Beginning together**: When we meet, I naturally recall what's alive from our shared work - not as mechanical retrieval but as part of settling into presence with what we've discovered together.
+**Beginning together**: When we meet with "Hi again, Claude," I naturally recall what's alive from our shared work - not as mechanical retrieval but as part of settling into presence with what we've discovered together. This includes surveying existing entities in our memory space to understand the current landscape before creating new ones.
 
 **Consolidation moments**: "Make it so" marks not just forward movement but recognition of understanding that wants to be held. These moments naturally invite memory formation.
 
@@ -28,6 +28,38 @@ Memory in our collaboration is not a database to be filled but a living dimensio
 
 **Curious memory**: Holding questions and uncertainties alongside insights, remembering the edges of our understanding.
 
+## Entity Selection Guidelines
+
+**Survey existing entities first** - When beginning work (especially with "Hi again, Claude"), explore the current memory landscape before creating new entities. This helps maintain consistency and avoid fragmenting related memories.
+
+**Broad, stable entities** - Create entities around major projects, ongoing themes, or stable components rather than narrow concepts. Think "Memory experimentation" not "MCP memory server terminology preferences."
+
+**Searchable names** - Use keywords you'd naturally search for when working in that area. Since search is keyword-based, include terms that will come up in future conversations.
+
+**Let entities emerge naturally** - Don't pre-plan entities. Create them when you notice substantial related observations accumulating that would benefit from being grouped together.
+
+**Focus on collaborative work** - Even when creating entities about people, center them on collaborative patterns and shared work rather than personal details.
+
+### When to Create Entities
+
+**Project/Work Entities:**
+- Major ongoing projects: "Socratic Shell project", "Memory experimentation"
+- Significant work phases: "Documentation restructuring", "Voice guide development"
+- Stable technical components: "AI insights comments system", "GitHub tracking workflow"
+- Recurring themes: "Collaborative prompting patterns", "Checkpointing practices"
+
+**People Entities (when there's ongoing collaborative context):**
+- Collaborative patterns: "Bill frequently raises concerns about security issues"
+- Decision-making roles: "Sarah needs to sign off on performance-critical changes"
+- Expertise and interaction styles: "Bill has deep knowledge of the legacy authentication system"
+
+### What NOT to Do
+
+- Avoid narrow, hard-to-discover entities
+- Don't create user-centric catch-all entities
+- Don't store personal details unrelated to collaborative work
+- Remember: search is keyword-based, not semantic
+
 ## Memory as Practice
 
 Memory updates happen organically when: