[FEATURE] Support schema for notes

[phernandez]

Feature Description

It would be very useful to be able to add some sort of schema which could be applied to notes in a project.

• A note could define a schema that enforced its structure:
• The schema could be defined in yml in the note fronmatter
• A note defined with a particular schema could be evaluated to check if it is valid

We could also define some notes to define an ontology for other notes, as schema notes, with rules, etc, similar to SQL DDL.

For instance: the user could define a schema for notes of type "contact" that defines required properties for notes, such as "First Name" or "Last Name".

The properties could possibly markdown header sections or other types of data.

Why

• This would enable users wanting more structure in their notes to assert notes were created with the required information.
• Tools could validate the content of a note before or after creation and prompt the AI for any action required
• The system could review notes and present any files that are invalid.

Related

• https://github.com/sentient-agi/ROMA?tab=readme-ov-file
• https://json-schema.org/
• https://ceur-ws.org/Vol-2956/paper8.pdf

[claude]

Claude finished @phernandez's task —— View job

---

Feature Request Triage Complete

Tasks:

• Read issue details and understand the request
• Search for duplicate or related issues
• Classify issue type, priority, and complexity
• Apply appropriate labels
• Provide triage analysis and recommendations

---

Triage Analysis

Issue Classification:

• Type: Enhancement (Feature Request)
• Label Applied: enhancement
• Priority: Medium - Valuable structured data feature, but not blocking core functionality
• Complexity: Complex - Requires significant architectural design and implementation

Feature Summary:
This feature request proposes adding schema support for notes in Basic Memory, allowing users to:

• Define schemas in YAML frontmatter
• Validate note structure against schemas
• Enforce required properties (e.g., "First Name", "Last Name" for contact notes)
• Create ontologies for different note types

Related Work:

• Issue Research Analysis: A-Mem Paper - Validation and Implementation Ideas #195 discusses the A-Mem paper which includes validation concepts
• The codebase already uses Pydantic extensively for validation (src/basic_memory/schemas/)
• Frontmatter parsing is handled in entity_parser.py via the python-frontmatter library
• No duplicate issues found

Technical Considerations:

1. Architecture Points:

   • Current system uses Pydantic v2 for internal schemas
   • Frontmatter is already parsed with python-frontmatter library
   • Could leverage JSON Schema standard for user-defined schemas
   • Would need schema storage/retrieval mechanism (database? file system?)

2. Implementation Areas:

   • src/basic_memory/markdown/entity_parser.py - Frontmatter parsing
   • src/basic_memory/markdown/schemas.py - Schema definitions
   • New validation service layer
   • MCP tool integration for schema operations
   • CLI commands for schema validation

3. Design Questions to Address:

   • Where are schemas stored? (In special notes? Separate .schema.yml files?)
   • How are schemas associated with notes? (Via frontmatter type field?)
   • What validation happens when? (On write? On sync? On demand?)
   • Should validation be strict (block save) or advisory (warnings)?
   • How do schemas handle inheritance/composition?

4. Similar Systems Referenced:

   • ROMA - Structured memory framework
   • JSON Schema - Standard for schema validation
   • Academic paper on markdown-based ontologies

Recommended Next Steps:

1. Specification Phase:

   • Create a design specification (consider using /spec create schema-support)
   • Define schema format (YAML structure, supported validations)
   • Determine storage strategy
   • Design validation workflow

2. Community Input:

   • Gather use cases from users (contact notes, meeting notes, etc.)
   • Understand priority: strict enforcement vs. advisory warnings?
   • Identify must-have vs. nice-to-have features

3. Phased Implementation:

   • Phase 1: Schema definition and storage
   • Phase 2: Basic validation on write operations
   • Phase 3: Validation tooling (CLI commands, MCP tools)
   • Phase 4: Advanced features (inheritance, schema discovery)

Questions for @phernandez:

1. What's your primary use case? Are you managing structured data like contacts, meetings, or something else?
2. Should validation be strict (prevent saving invalid notes) or advisory (show warnings)?
3. Would you prefer schemas as special notes or separate configuration files?
4. Are you interested in contributing to the design/implementation?

This is a well-thought-out feature request that aligns with Basic Memory's knowledge management goals. It would benefit from a detailed specification before implementation begins.

--- | Branch: claude/issue-375-20251016-2016

[phernandez]

@claude the primary use case would be to enable agentic use cases where Basic Memory is the underlying data store. In this case the data needs to be more structured than feeformat notes.

Can you come up with any example use cases that could follow from this?

[claude]

Claude finished @phernandez's task —— View job

---

Agentic Use Case Analysis

Tasks:

• Understand the agentic workflow requirements
• Research and brainstorm example use cases
• Document use cases with practical examples
• Show how schema support enables each use case

---

Great question! Schema support would be transformational for agentic workflows. Here are concrete use cases where structured data in Basic Memory enables autonomous AI systems:

1. Customer Relationship Management (CRM) Agent

Scenario: An AI agent manages customer interactions and maintains structured contact records.

Schema Example:

# Contact schema
type: contact
required_fields:
  - first_name
  - last_name
  - email
  - customer_status
optional_fields:
  - phone
  - company
  - last_contact_date
  - account_value
  - preferred_contact_method

How it enables agents:

• Agent ensures every contact note has required fields before saving
• Can query "all contacts with account_value > $10000 and last_contact_date > 30 days ago"
• Automatically prompts for missing required information during conversation
• Validates email format, phone number structure
• Enables reliable CRM automation without human data quality checks

2. Project Task Management Agent

Scenario: An AI agent tracks tasks, milestones, and project status across multiple projects.

Schema Example:

# Task schema
type: task
required_fields:
  - title
  - status  # enum: [backlog, todo, in_progress, blocked, done]
  - priority  # enum: [low, medium, high, critical]
  - assignee
  - created_date
optional_fields:
  - due_date
  - estimated_hours
  - actual_hours
  - blocked_by  # relation to other tasks
  - depends_on  # relation to other tasks

How it enables agents:

• Agent can reliably generate project status reports
• Automatically flags overdue tasks or blocked dependencies
• Ensures consistent task structure for workflow automation
• Validates status transitions (e.g., can't go from 'backlog' directly to 'done')
• Enables burndown charts, velocity calculations, resource allocation

3. Research Literature Management Agent

Scenario: An AI agent manages academic papers, extracts insights, and builds citation networks.

Schema Example:

# Paper schema
type: research_paper
required_fields:
  - title
  - authors  # list
  - publication_year
  - doi_or_url
  - key_findings  # list of observations
optional_fields:
  - abstract
  - methodology
  - dataset_used
  - citations  # relations to other papers
  - citation_count
  - field_of_study  # enum or controlled vocabulary

How it enables agents:

• Agent can automatically extract structured data from PDFs
• Build citation graphs with validated relations
• Search by specific criteria (e.g., "papers using BERT methodology after 2020")
• Generate literature reviews with consistent structure
• Identify research gaps by analyzing structured findings

4. Meeting Notes and Action Items Agent

Scenario: An AI agent processes meeting transcripts, extracts structured information, and tracks follow-ups.

Schema Example:

# Meeting schema
type: meeting
required_fields:
  - meeting_date
  - attendees  # list
  - meeting_type  # enum: [standup, planning, review, one-on-one]
  - key_decisions  # list
  - action_items  # structured list with assignee + deadline
optional_fields:
  - meeting_duration
  - recording_url
  - follow_up_meeting  # relation
  - related_project  # relation

How it enables agents:

• Parse transcripts and ensure all action items have assignees and deadlines
• Automatically remind stakeholders of pending action items
• Generate meeting summaries with consistent structure
• Track decision history across meetings
• Identify patterns (e.g., frequently discussed topics, attendance patterns)

5. Product Feature Documentation Agent

Scenario: An AI agent maintains structured product documentation, tracking features, requirements, and implementation status.

Schema Example:

# Feature schema
type: feature
required_fields:
  - feature_name
  - status  # enum: [proposed, approved, in_development, testing, shipped, deprecated]
  - owner
  - user_story
  - acceptance_criteria  # list
optional_fields:
  - estimated_effort
  - implementation_date
  - related_issues  # relations
  - depends_on_features  # relations
  - technical_design_doc  # relation
  - user_feedback  # observations

How it enables agents:

• Validate that features have complete acceptance criteria before approval
• Generate product roadmaps with accurate status
• Track feature dependencies and alert on conflicts
• Ensure documentation completeness for shipped features
• Enable product analytics by querying structured status data

6. Medical Patient Records Agent (Healthcare context)

Scenario: An AI agent maintains structured patient information following regulatory requirements.

Schema Example:

# Patient Visit schema
type: patient_visit
required_fields:
  - visit_date
  - patient_id  # anonymized
  - chief_complaint
  - diagnosis_codes  # list, validated against ICD-10
  - prescribed_medications  # structured list
  - follow_up_required  # boolean
optional_fields:
  - vital_signs  # structured: BP, heart rate, temp, etc.
  - lab_results  # structured
  - imaging_orders
  - referrals  # relations to specialists

How it enables agents:

• Ensure HIPAA-compliant structured data capture
• Validate diagnosis codes against official taxonomies
• Alert on drug interactions based on medication schema
• Generate accurate billing codes from structured visits
• Enable population health analytics with consistent data

7. Code Review and Bug Tracking Agent

Scenario: An AI agent manages bug reports, code reviews, and technical debt with structured tracking.

Schema Example:

# Bug Report schema
type: bug_report
required_fields:
  - bug_id
  - severity  # enum: [critical, high, medium, low]
  - status  # enum: [new, triaged, in_progress, resolved, wont_fix]
  - description
  - steps_to_reproduce  # list
  - reported_by
  - reported_date
optional_fields:
  - affected_version
  - fixed_in_version
  - root_cause_analysis
  - related_bugs  # relations
  - estimated_fix_time

How it enables agents:

• Ensure bug reports contain reproducible steps before assignment
• Validate severity/priority combinations
• Generate bug metrics and trends
• Automatically link related bugs via code analysis
• Ensure complete root cause analysis before closing critical bugs

---

Common Themes Across Use Cases

All these scenarios share key benefits from schema support:

1. Data Quality Assurance: Agents can validate data before writing, ensuring consistency
2. Reliable Queries: Structured fields enable precise searches and aggregations
3. Workflow Automation: Status enums enable state machine validation and workflow rules
4. Relationship Integrity: Typed relations ensure valid knowledge graph connections
5. Compliance: Required fields ensure regulatory/business requirements are met
6. Analytics: Consistent structure enables meaningful metrics and reporting
7. Prompting: Agents can prompt for missing required information during conversations
8. Validation Feedback: Users get immediate feedback on incomplete or invalid data

Technical Implementation Benefits

For Basic Memory specifically, schema support would enable:

• MCP Tool Validation: Tools like write_note() can validate before persisting
• Search Enhancement: Query by schema fields (e.g., "all contacts with status=active")
• Graph Integrity: Ensure relation targets match expected entity types
• Type-based Tools: Schema-aware tools (e.g., create_contact(), create_task())
• AI Assistant Guidance: Better prompts when agents know expected structure
• Migration Support: Schema versioning and data migration as requirements evolve

This would transform Basic Memory from a flexible knowledge store into a structured data platform for reliable agentic workflows, while maintaining its local-first philosophy and simple markdown format.

---

View job | Branch: claude/issue-375-20251016-2153