-
Notifications
You must be signed in to change notification settings - Fork 148
fix: preserve search index across server restarts #503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Claude finished @phernandez's task in 2m 43s —— View job PR Review Complete ✓I've reviewed this PR against the Basic Memory team checklist. This is a critical bug fix that resolves a data loss issue affecting v0.16.3-v0.17.3. Code Quality & Standards ✓
Testing & Documentation ✓
However: Missing test for the specific bug being fixed (see recommendations below) Basic Memory Architecture ✓
Security & Performance ✓
Compatibility ✓
AnalysisRoot Cause Confirmed: The Fix Verification:
User Impact: Users experienced intermittent search failures because Claude Desktop periodically restarts the MCP server (every ~30 minutes), wiping the index. Recommendations1. Add Regression Test (Important)While existing tests verify @pytest.mark.asyncio
async def test_init_search_index_preserves_existing_data(search_repository, search_entity):
"""Test that re-initializing search index preserves existing indexed data."""
# Index an entity
search_row = SearchIndexRow(
project_id=search_entity.project_id,
id=search_entity.id,
permalink=search_entity.permalink,
file_path=search_entity.file_path,
type=SearchItemType.ENTITY.value,
content_stems="test content",
)
await search_repository.index_item(search_row)
# Verify it's searchable
results = await search_repository.search(search_text="test")
assert len(results) == 1
# Re-initialize the search index (simulates server restart)
await search_repository.init_search_index()
# Verify data is still there
results_after = await search_repository.search(search_text="test")
assert len(results_after) == 1
assert results_after[0].id == search_entity.idLocation: This test would have caught the bug and prevents regression. 2. Manual Verification ChecklistThe PR description shows manual verification as incomplete:
For completeness, verify:
SummaryVerdict: ✅ Approved - Critical bug fix with excellent root cause analysis This PR:
Only concern: Missing regression test to prevent this bug from reoccurring. Since this is a critical fix affecting users now, I recommend:
The fix itself is sound and well-documented. |
Remove unconditional DROP TABLE from init_search_index() that was wiping the FTS5 search index on every MCP server startup. The bug was introduced in PR #439 (v0.16.3) when splitting SQLite and Postgres search implementations. The DROP TABLE was added with a comment about Base.metadata.create_all() potentially creating a regular table, but this caused all indexed data to be lost whenever Claude Desktop restarted the MCP server. Symptoms users experienced: - Notes searchable immediately after creation - Search returns empty results after ~30 minutes (server restart) - Files exist on disk but search index is empty - recent_activity() shows "No recent activity" The fix simply removes the DROP TABLE since CREATE_SEARCH_INDEX already uses "CREATE VIRTUAL TABLE IF NOT EXISTS" which safely handles both cases (table exists vs doesn't exist). Affected users should run `basic-memory reset` once after updating to rebuild their search index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> Signed-off-by: phernandez <[email protected]>
- Skip database initialization for reset command (like mcp/status/etc) - Delete WAL files (-shm, -wal) along with main database file - Add user-friendly error messages when database is locked - Show reassuring message that note files won't be deleted 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> Signed-off-by: phernandez <[email protected]>
- Add run_in_background parameter to run_sync() for foreground sync - Handle both background and foreground response formats - Refactor reset command to use get_sync_service() directly - Avoid ASGI transport which caused process hang on completion - Ensure proper database cleanup with db.shutdown_db() - Remove unused imports in test files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> Signed-off-by: phernandez <[email protected]>
9acd2a6 to
f2d906d
Compare
|
Claude finished @phernandez's task in 3m 8s —— View job PR Review Complete ✓I've reviewed this updated PR (now 3 commits) against the Basic Memory team checklist. The PR has evolved from a single critical bug fix to include important improvements to the Summary of ChangesCommit 1:
|
| File | Lines | Purpose |
|---|---|---|
| sqlite_search_repository.py | +4/-6 | Critical fix - Remove DROP TABLE, preserve index |
| cli/commands/db.py | +78/-19 | Improve reset UX, fix process hang, add foreground sync |
| cli/commands/command_utils.py | +25/-3 | Add run_in_background support, foreground sync output |
| cli/app.py | +3/-2 | Skip init for reset command |
| test_clients.py | +1/-1 | Remove unused import |
| test_project_resolver.py | +0/-1 | Remove unused import |
Add test_init_search_index_preserves_data() to verify that calling init_search_index() twice preserves indexed data, simulating MCP server restart behavior. This test would have caught the bug fixed in PR #503 where init_search_index() was unconditionally dropping the search_index table on every call. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> Signed-off-by: phernandez <[email protected]>
|
Claude finished @phernandez's task in 1m 54s —— View job PR Review Complete ✓I've reviewed this PR (4 commits, 7 files changed) against the Basic Memory team checklist. This PR has significantly evolved from the original critical bug fix and now includes important improvements to the Summary of ChangesCommit 1:
|
| File | Lines | Purpose | Status |
|---|---|---|---|
| sqlite_search_repository.py | +4/-6 | Critical fix - Remove DROP TABLE, preserve index | ✅ |
| cli/commands/db.py | +78/-19 | Improve reset UX, fix process hang, add foreground sync | ✅ |
| cli/commands/command_utils.py | +25/-3 | Add run_in_background support, cleanup helper | ✅ |
| cli/app.py | +3/-2 | Skip init for reset command | ✅ |
| test_search_repository.py | +42/-0 | Regression test for index persistence | ✅ |
| test_clients.py | +1/-1 | Remove unused import | ✅ |
| test_project_resolver.py | +0/-1 | Remove unused import | ✅ |
Total: +153/-32 lines across 7 files
- Critical bug fix for search index preservation (#503) - Major architecture refactor with composition roots (#502) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> Signed-off-by: phernandez <[email protected]>
Summary
DROP TABLEfrominit_search_index()that was wiping the FTS5 search index on every MCP server startupCREATE_SEARCH_INDEXDDL already usesCREATE VIRTUAL TABLE IF NOT EXISTS, so the DROP was unnecessary and destructiveProblem
Users reported that search would work immediately after creating notes, but return empty results after ~30 minutes. The pattern was:
write_note()→ immediately searchable ✓Root cause: Every time Claude Desktop restarts the MCP server,
init_search_index()was called during migrations, which dropped and recreated an empty search index.Fix
Simply remove the
DROP TABLE IF EXISTS search_indexline. TheCREATE VIRTUAL TABLE IF NOT EXISTShandles both cases safely.Affected Versions
User Action Required
After updating, users should run
basic-memory resetonce to rebuild their search index from existing markdown files.Test plan
🤖 Generated with Claude Code