Section and Observation Anchor Wikilinks Support

[calebjpicker]

Feature Description

Add support for anchor links within documents, allowing wikilinks to reference specific sections or elements within a markdown file. Currently, Basic Memory only supports linking to entire documents via title, permalink, or memory URL.

Problem This Feature Solves

When working with long documents (meeting notes, research papers, conference notes), I need to reference specific quotes, insights, or sections. Currently I must either:

• Create separate files for each quote (breaks context)
• Link to entire documents (reader must search for relevant part)
• Rely on search (doesn't create semantic connections)

Proposed Solution

Enable two types of anchor links:

1. Section anchors: [[Meeting Notes#Project-Updates]] - links to ## Project Updates
2. Observation anchors: [[Meeting Notes#Project-Updates#id-0001]] - links to any element tagged with #id-0001 within that section

Proposed Syntax:
[[entity#section#observation]]

Where:

• entity = document name/title
• section = heading within document (optional)
• observation = specific element ID (optional)

Example workflow:

# Team Meeting 2025-06-14

## Budget Discussion
[quote] "We need to triple our marketing spend" (marketing) #id-0001 #john
[quote] "Last year's ROI was only 1.2x" (marketing) #id-0002 #sarah
  • based_on [[Q1 Marketing Report 2025#roi-analysis#figure-3]]
[action] Review Q1 marketing metrics before decision (marketing) #id-0003

## Tech Updates  
[issue] Accounting system can't handle multi-currency (accounting) #id-0004
[finding] $50k loss from exchange rate errors last quarter (accounting) #id-0005 #teri
  • discovered_in [[Financial Audit 2025#exchange-rate-errors#table-2]]
[todo] Schedule vendor demos for new system (tech) #id-0006

## Relations
- relates_to [[Q1 Marketing Performance Report]]
- impacts [[International Expansion Strategy#europe-section]]
- contradicts [[2024 Budget Decisions#marketing#spend-reduction]]

What This Feature Would Enable:

Basic Memory currently supports:
• [[Team Meeting]] → relates_to → [[Marketing Report]] ✓
(whole document → whole document)

With anchor support, we could use flexible syntax for any granularity:

Between Documents:
• [[Budget Analysis]] → based_on → [[Team Meeting#Budget-Discussion]]
(entity → section of another entity)

• [[Marketing Report#roi-analysis]] → contradicts → [[Team Meeting#Budget-Discussion#id-0001]]
(section → specific observation in another doc)

• [[Team Meeting#Budget-Discussion#id-0002]] → derived_from → [[Marketing Report#roi-analysis#figure-3]]
(observation → observation across documents)

Within the Same Document:
• [[#Summary]] → summarizes → [[#Budget-Discussion]]
(section → section in same doc)

• [[#Tech-Updates#id-0004]] → caused_by → [[#Tech-Updates#id-0005]]
(observation → observation in same doc)

• [[#Action-Items]] → implements → [[#Budget-Discussion#id-0001]], [[#Budget-Discussion#id-0003]], [[#Tech-Updates#id-0006]]
(section → multiple observations across sections in same doc)

The syntax supports any combination:

• [[entity]] - whole document
• [[entity#section]] - specific section
• [[entity#section#observation]] - specific element
• [[#section]] - section in current doc
• [[#section#observation]] - specific element in current doc

In analysis documents:

# Q2 Analysis

John proposed [[Team Meeting 2025-06-14#Budget-Discussion#id-0001]] increasing spend, 
but Sarah noted [[Team Meeting 2025-06-14#Budget-Discussion#id-0002]] poor historical ROI.

Alternative Solutions

• Creating one file per observation (too granular)
• Using very descriptive filenames (still links to whole document)
• Manual search for tags (no semantic graph connection)

Additional Context

This would enhance existing Basic Memory workflows like:

• Conference Notes: Link to specific speaker quotes
• Research Papers: Reference exact findings or methods
• Meeting Notes: Connect decisions to who said what
• Daily Reflections: Build precise knowledge chains

Without anchors, a "Weekly Planning" document can only say "see Tuesday's meeting notes" instead of linking directly to the specific decision or quote that matters.

Impact

This feature would:

• Enable precise citation networks
• Reduce need for file fragmentation
• Make knowledge graphs more granular and useful
• Support evidence-based workflows
• Enhance the AI's ability to follow specific references when building context

[phernandez]

@calebjpicker thanks for the suggestions. I like this idea. I'd have to give some thought to how it might work. Currently a link (Relation) is just to another note (Entity). We'd probably have to store more metadata in the db for all of the headings in a note.

[phernandez]

@claude can you think of a few ideas to implement the user's suggestion here? What are some approaches we could take?

[claude]

Claude finished @phernandez's task —— View job

---

Analysis: Anchor Wikilinks Implementation Ideas

I've analyzed the current codebase and can propose several approaches for implementing anchor link support. Here's my comprehensive analysis:

Todo List:

• Analyze current relation/linking system
• Review markdown parsing and entity storage
• Propose database schema approaches
• Design anchor parsing strategy
• Consider navigation and resolution methods
• Address performance and indexing implications
• Provide implementation recommendations

---

Current System Analysis

Current Architecture:

• Entities: Markdown files with id, title, permalink, file_path
• Relations: Links between entities with from_id, to_id, to_name, relation_type
• Observations: Facts about entities with content, category, tags
• Navigation: Uses memory:// URLs and permalinks for entity-level access
• Parsing: markdown-it plugins extract [[target]] links from content

Current Limitations:

• Relations only support whole-entity linking (to_id points to entire Entity)
• No support for section headers or observation-level anchors
• Memory URLs only resolve to entity level: memory://entity-name

---

Implementation Approaches

Approach 1: Enhanced Relation Model

Extend existing relations to support anchor targeting

Database Changes:

-- Add anchor support to existing relation table
ALTER TABLE relation ADD COLUMN target_section VARCHAR(255);
ALTER TABLE relation ADD COLUMN target_observation_id INTEGER;
ALTER TABLE relation ADD COLUMN target_anchor VARCHAR(255); -- for #id-0001 style anchors

Pros:

• Minimal schema changes
• Leverages existing relation infrastructure
• Backward compatible

Cons:

• Makes relation table more complex
• Target resolution becomes more complicated
• No direct indexing of sections/observations as linkable entities

---

Approach 2: Anchor Registry Table

Create a separate table to index all linkable anchors

Database Schema:

-- New table for indexable anchors
CREATE TABLE anchor (
    id INTEGER PRIMARY KEY,
    entity_id INTEGER NOT NULL,
    anchor_type VARCHAR(50) NOT NULL, -- 'section', 'observation', 'custom'
    anchor_id VARCHAR(255) NOT NULL,  -- section name or observation ID
    content TEXT,                     -- heading text or observation content
    line_number INTEGER,             -- location in file
    created_at TIMESTAMP,
    FOREIGN KEY (entity_id) REFERENCES entity(id),
    UNIQUE(entity_id, anchor_id)
);

-- Enhanced relations table
ALTER TABLE relation ADD COLUMN target_anchor_id INTEGER;
ALTER TABLE relation ADD FOREIGN KEY (target_anchor_id) REFERENCES anchor(id);

Pros:

• Clean separation of concerns
• Anchors become first-class linkable entities
• Enables rich anchor-based navigation and search
• Supports advanced use cases (anchor-to-anchor relations)

Cons:

• More complex schema
• Requires anchor extraction during sync
• Additional storage overhead

---

Approach 3: JSON Column Extension

Store anchor information as JSON metadata

Database Changes:

-- Add JSON column to relations
ALTER TABLE relation ADD COLUMN target_metadata JSON;

-- Example data structure:
-- {"section": "Budget-Discussion", "observation": "id-0001", "anchor_type": "observation"}

Pros:

• Flexible schema for different anchor types
• Easy to extend with new anchor metadata
• Single column addition

Cons:

• JSON queries are less efficient than relational
• No referential integrity for anchors
• Harder to index and search

---

Recommended Approach: Hybrid Approach 2 + Enhancements

I recommend Approach 2 (Anchor Registry) with these enhancements:

Database Schema:

-- Anchor registry for all linkable elements
CREATE TABLE anchor (
    id INTEGER PRIMARY KEY,
    entity_id INTEGER NOT NULL,
    anchor_type VARCHAR(50) NOT NULL, -- 'heading', 'observation', 'custom'
    anchor_key VARCHAR(255) NOT NULL, -- normalized anchor identifier
    display_text TEXT NOT NULL,       -- human-readable text
    content_hash VARCHAR(64),         -- for change detection
    parent_anchor_id INTEGER,         -- for nested sections
    line_start INTEGER,
    line_end INTEGER,
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    FOREIGN KEY (entity_id) REFERENCES entity(id) ON DELETE CASCADE,
    FOREIGN KEY (parent_anchor_id) REFERENCES anchor(id) ON DELETE CASCADE,
    UNIQUE(entity_id, anchor_key)
);

-- Enhanced relation table
ALTER TABLE relation ADD COLUMN target_anchor_id INTEGER;
ALTER TABLE relation ADD FOREIGN KEY (target_anchor_id) REFERENCES anchor(id) ON DELETE SET NULL;

-- Indexes for performance
CREATE INDEX idx_anchor_entity_type ON anchor(entity_id, anchor_type);
CREATE INDEX idx_anchor_key ON anchor(entity_id, anchor_key);
CREATE INDEX idx_relation_target_anchor ON relation(target_anchor_id);

Parsing Strategy:

1. Section Anchors: Extract from markdown headers

   ## Project Updates  → anchor_key: "project-updates"
   ### Q1 Results     → anchor_key: "q1-results", parent: "project-updates"

2. Observation Anchors: Extract from tagged observations

   [quote] "Triple marketing spend" #id-0001 → anchor_key: "id-0001"

3. Link Parsing: Update markdown plugin to handle new syntax

   [[Meeting Notes#Budget-Discussion#id-0001]]
   ↓
   entity: "Meeting Notes"
   section: "Budget-Discussion" 
   observation: "id-0001"

Memory URL Extensions:

memory://entity-name                    # Current: whole entity
memory://entity-name#section            # New: section anchor  
memory://entity-name#section#obs-id     # New: observation anchor
memory://entity-name#obs-id             # New: direct observation

---

Performance & Indexing Analysis

Storage Impact:

• Anchor Table: ~100-200 bytes per anchor
• Relation Extensions: ~8 bytes per relation (one INTEGER column)
• Overall Overhead: 20-30% increase for typical knowledge bases
• Acceptable Trade-off: Precision gains justify storage cost

Query Performance:

• Anchor Lookups: O(1) with proper indexing on (entity_id, anchor_key)
• Relation Traversal: Minimal impact - single JOIN to anchor table
• Memory URL Resolution: 2-3 queries max (entity → anchor → content)
• Bulk Operations: Anchor extraction parallelizable during sync

Indexing Strategy:

-- Core performance indexes
CREATE INDEX idx_anchor_lookup ON anchor(entity_id, anchor_key);
CREATE INDEX idx_anchor_content ON anchor(anchor_type, display_text);
CREATE INDEX idx_relation_anchored ON relation(target_anchor_id) WHERE target_anchor_id IS NOT NULL;

-- Full-text search integration
-- Could extend existing FTS to include anchor content

---

Implementation Plan

Phase 1: Database Schema (2-3 days)

• Create anchor table with Alembic migration
• Add target_anchor_id to relation table
• Update SQLAlchemy models (Entity, Relation, new Anchor model)
• Add repository methods for anchor CRUD

Phase 2: Parsing Enhancement (3-4 days)

• Extend markdown plugins for anchor extraction
• Update entity parser to build anchor registry during sync
• Modify relation parsing for [[entity#section#observation]] syntax
• Add anchor diff/sync logic for change detection

Phase 3: Navigation & Resolution (2-3 days)

• Extend memory URL parsing for anchors (memory://entity#section)
• Update read_note tool for anchor navigation
• Enhance build_context for anchor-aware traversal
• Add anchor resolution to API routers

Phase 4: MCP Tool Updates (2-3 days)

• Update write_note to support anchor creation
• Extend search to include anchor content
• Add anchor-specific navigation tools
• Update view_note for anchor highlighting

Phase 5: Testing & Documentation (2-3 days)

• Comprehensive test suite for anchor functionality
• Update CLI tools and prompts
• Documentation and usage examples
• Performance benchmarking

Total Estimated Effort: 11-16 days

---

Alternative: Simpler String-Based Approach

If the full anchor registry seems too complex, here's a lighter alternative:

Minimal Schema Changes:

-- Just extend relations with anchor strings
ALTER TABLE relation ADD COLUMN target_section VARCHAR(255);
ALTER TABLE relation ADD COLUMN target_anchor VARCHAR(255);

Runtime Resolution:

• Parse anchor references at query time
• No persistent anchor registry
• Lighter storage footprint
• Trade-off: Less rich navigation, no anchor-as-entity features

---

Key Benefits of Full Implementation

1. Precise Citations: [[Meeting#Budget#id-0001]] → exact quote
2. Semantic Granularity: Relations between specific observations
3. Enhanced AI Context: Claude can follow exact references
4. Flexible Syntax: [[#section]] for same-document references
5. Future-Proof: Anchor registry enables advanced features

Conclusion

The Anchor Registry approach provides the best balance of:

• Functionality: Full anchor link support as requested
• Performance: Efficient with proper indexing
• Extensibility: Foundation for future enhancements
• Maintainability: Clean separation of concerns

The implementation is substantial but manageable, delivering significant value for knowledge management workflows.

---

[calebjpicker]

@phernandez Awesome! I see you had claude think through some possible ways to implement it. How exciting!

Between- and within- entity x section x observation level granularity would be incredible, especially for researchers in academia or legal contexts.

[phernandez]

@calebjpicker Yeah, I think that sounds really cool. I'll see if I can work on it soon.