Automatic RAG-like discovery of insights

[nikomatsakis]

Automatic RAG-like Discovery of Insights

Status: In Progress

Current Understanding

Implement a conversation monitoring system that automatically surfaces relevant insights without explicit search commands. The system should:

1. Track conversational context - maintain a rolling 2-3 sentence summary of "what the user is doing right now"
2. Automatic background search - use user prompts + situational context to search for relevant insights
3. Ambient memory integration - surface insights seamlessly before AI responses

This would make the memory system feel intelligent and contextually aware, like having a research assistant who's always paying attention.

Implementation Approach

Phase 1: Simple Case (Current Focus)

• Use working directory hash as conversation ID for persistence
• Integrate ollama-rs crate for local LLM communication (Llama 3.2 1B)
• Implement conversation tracker with persistent state in ~/.hippo/conversations/
• Add hippo_summarize_context MCP tool for testing and debugging
• Hook into MCP message processing to update context on user messages

Phase 2: Session Lifecycle (Later)

• Use session start hooks to handle quit/resume workflow
• Clear conversation state on session start, otherwise resume from last state
• Handle context transitions and topic shifts gracefully

Next Steps

• Add ollama-rs dependency to Cargo.toml
• Implement basic ConversationTracker struct with persistent state
• Add hippo_summarize_context MCP tool for testing
• Hook conversation updates into message processing pipeline
• Test with simple conversation scenarios

Open Questions

• What's the optimal context window for situational tracking?
• How to balance automatic search frequency vs performance overhead?
• Should we fall back to embedding-based approaches if Ollama is unavailable?
• How to tune the LLM prompts for consistent context summarization?

Context

Current memory system requires explicit search commands, making it feel disconnected from natural conversation flow. Automatic discovery would create a more seamless experience where relevant insights surface naturally based on what the user is actually working on.

The temporal scoring system is now complete, providing a solid foundation for intelligent insight ranking. This feature builds on that to make memory retrieval truly ambient.

Progress

• ✅ Completed temporal scoring implementation (issues Port Hippo to Rust for improved startup performance #6, Port Python Temporal Scoring System to Rust #8)
• ✅ Researched local LLM options and conversation persistence approaches
• ✅ Defined implementation strategy focusing on simple case first
• 🔄 Ready to begin Phase 1 implementation

[nikomatsakis]

Session summary:
Explored multiple approaches for local LLM integration but discovered fundamental adoption barriers that require rethinking the implementation strategy.

Progress made:

• ✅ Implemented ConversationTracker with persistent state using working directory hash as conversation ID
• ✅ Created comprehensive integration tests for conversation tracking scenarios
• ✅ Successfully tested Q CLI integration as alternative to Ollama (using q chat --no-interactive --model claude-3.7-sonnet)
• ✅ Explored direct model loading with candle framework for Qwen2.5 models (0.5B and 1.5B variants)

Key discovery - User adoption barrier:
The original Ollama approach requires every user to:

1. Install Ollama (~100MB+ download)
2. Download 1.3GB+ model files
3. Run a local LLM service
4. Have sufficient RAM/CPU for inference

This creates a massive barrier to adoption that makes the feature unusable for most users.

Approaches explored:

1. Q CLI integration (hacky but functional):

   • Pros: Uses existing Q infrastructure, no additional dependencies, high-quality summaries
   • Cons: Very slow (3-5 seconds per summary), process startup overhead
   • Status: Working prototype with good quality results

2. Direct model loading with candle:

   • Pros: No external dependencies, lazy download, configurable model sizes
   • Cons: Hit dependency conflicts with rand crates, candle ecosystem still evolving
   • Status: Compilation failures due to version conflicts

3. Mock implementation:

   • Created pattern-matching mock for testing infrastructure
   • Useful for CI/CD but defeats the purpose of evaluating actual LLM quality

Impact on approach:
The user adoption barrier is a fundamental architectural issue that needs to be resolved before implementation details. The feature needs to either:

• Be optional with graceful degradation
• Use remote APIs instead of local models
• Provide multiple backends (local LLM, remote API, heuristics)
• Find a way to make local models truly zero-setup

Next steps:
Reverting current implementation to start fresh with a cleaner approach that addresses the adoption barrier from the beginning. Will explore making the LLM dependency optional or finding lighter-weight alternatives that don't require users to install additional infrastructure.

[nikomatsakis]

Session summary:
Successfully implemented the foundation for ONNX Runtime-based conversation tracking, replacing the Ollama dependency with direct model loading approach.

Progress made:

• ✅ Added ort 2.0.0-rc.10 and tokenizers dependencies with proper API structure based on GPT2 example
• ✅ Created LocalLLM struct using ort session patterns instead of HTTP calls to Ollama
• ✅ Implemented ConversationTracker with ort integration and comprehensive test coverage
• ✅ Replaced hash-based conversation IDs with human-readable format: _Users_nikomat_dev_hippo_rs#2025-08-03T11:40:10+0000#0
• ✅ Implemented path sanitization (/ → _, special chars percent-encoded) and atomic counter logic
• ✅ All tests passing, infrastructure ready for actual model loading

Impact on approach:
The user adoption barrier from the previous Ollama approach led us to pivot to direct ONNX model loading. This eliminates the need for users to install and run Ollama as a separate service, while still providing local inference capabilities. The ort approach is more lightweight and integrates directly into the Hippo binary.

Key architectural improvements:

• Replaced ollama-rs dependency with ort and tokenizers for direct model loading
• LocalLLM now uses ONNX Runtime sessions with proper API structure from ort examples
• Conversation IDs are human-readable and debuggable with atomic counter logic (prevents race conditions)
• Test-driven development approach ensures solid foundation before model loading implementation

Next steps:

• Implement actual ONNX model downloading and loading using ort session builder
• Set up tokenizer loading from model files (likely from Hugging Face model repos)
• Add basic text generation inference following ort patterns
• Test with real conversation summarization scenarios

The foundation is now solid and ready for actual model integration. All placeholder logic is clearly marked and the API structure matches proven ort patterns.

[nikomatsakis]

Progress Update: Robust Model Download System

Successfully implemented a production-ready model download system with integrity verification and atomic operations.

Key Achievements:

• ✅ Dedicated download module - Extracted complex download logic into for better organization
• ✅ SHA-256 verification - Downloads and verifies files from Hugging Face for integrity checking
• ✅ Memory-efficient streaming - Handles 1.5GB+ models without loading into memory using chunk-based processing
• ✅ Atomic operations - Uses system temp files with atomic rename to prevent corrupted partial downloads
• ✅ Comprehensive error handling - Proper cleanup on failures, graceful fallback when checksums unavailable
• ✅ System temp directory - Uses for better permissions and automatic cleanup
• ✅ Progress tracking - Logs download progress for large files with size information

Technical Implementation:

• Uses crate for proper temporary file management
• Calculates SHA-256 hash during streaming to avoid re-reading files
• Supports both "hash only" and "hash filename" checksum formats
• Includes comprehensive test coverage for edge cases
• Clean build with no lint warnings

Download Process:

1. Attempt to download checksum file (graceful fallback if missing)
2. Stream download to system temp file while calculating hash
3. Verify hash matches expected value (if checksum available)
4. Atomically rename temp file to final location
5. Clean up temp files automatically on any failure

Next Steps:

• Test with actual Qwen2.5 model downloads from Hugging Face
• Implement proper text generation logic using downloaded models
• Add conversation summarization with real ONNX inference

The download infrastructure is now robust and ready for production use with large language models.

[nikomatsakis]

Major Progress Update: ONNX Inference Exploration Complete

Successfully completed comprehensive exploration of ONNX Runtime integration for local model inference. This work has been preserved in a feature branch for future reference.

🎉 Achievements:

• ✅ Robust download system: SHA-256 verification, streaming, atomic operations with system temp files
• ✅ Real model loading: Successfully downloaded and loaded 500MB Qwen2.5-0.5B ONNX model
• ✅ Infrastructure validation: Proven that our download and caching approach works with real HuggingFace models
• ✅ Comprehensive research: Detailed analysis of Qwen2.5 requirements and implementation complexity

🔍 Key Discovery - Significant Complexity:
Text generation with Qwen2.5 ONNX models requires:

• 48 separate tensor inputs (24 layers × 2 tensors each for past_key_values)
• Grouped Query Attention with specific tensor shapes (2 KV heads, not 14)
• Iterative generation loops with attention mask management
• Complex state management for past key-value caching

This is essentially building a full transformer inference engine, not simple model usage.

🏗️ Architectural Decision:
Based on complexity analysis, we're separating inference logic into a distinct project:

• Hippo core: Focus on conversation tracking, insight storage, memory management
• Separate inference project: Handle all ONNX/transformer complexity with clean HTTP API
• Benefits: Cleaner interfaces, focused development, reusable inference library

📦 Work Preservation:
All ONNX exploration work has been preserved in feature branch:

• Branch: feature/onnx-inference-exploration
• Location: nikomatsakis/hippo fork
• Includes: Complete download system, model loading, research documentation, integration tests

🔄 Next Steps:

1. Revert main branch to focus on Hippo's core mission
2. Implement simple LLM client interface with mock responses
3. Focus on conversation tracking features that provide immediate value
4. Create separate inference project later using preserved work as foundation

📊 Impact Assessment:
This exploration was valuable - we now have:

• Proven download infrastructure (reusable)
• Deep understanding of inference complexity
• Clear architectural path forward
• Solid foundation for future inference project

The work represents successful proof-of-concept and informed architectural decision-making.