Skip to content

Conversation

@prosdev
Copy link
Collaborator

@prosdev prosdev commented Dec 3, 2025

Summary

Fixes the existing update() method to actually work. The infrastructure was already there, we just needed to wire it up.

Changes

1. Implement delete in vector store

  • Added delete() using LanceDB SQL predicate (id IN (...))
  • Properly escapes IDs to prevent SQL injection

2. Add new file detection

  • detectChangedFiles() now returns { changed, added, deleted }
  • Scans for files not in state to detect new files
  • Handles deleted files (in state but not on disk)

3. Update method improvements

  • Deletes documents for removed files
  • Deletes old documents before re-indexing changed files
  • Cleans up orphaned documents when symbols are removed from files
  • Properly updates state after deletions

Test Results

Test Files  70 passed | 1 skipped (71)
Tests  1490 passed | 11 skipped (1501)

Usage

# First run: full index
dev index .

# Subsequent runs: incremental (only changed/new files)
dev update

# Force full re-index
dev index . --force

Closes #122

- Implement delete() in LanceDB store using SQL predicate
- Add new file detection to detectChangedFiles()
- Handle deleted files in update() method
- Clean up orphaned documents when symbols are removed
- Add comprehensive tests for incremental update scenarios

Closes #122
@prosdev prosdev force-pushed the feat/122-incremental-indexing branch from 579d8f5 to bece9e3 Compare December 3, 2025 18:22
@prosdev prosdev merged commit 27f87cc into main Dec 3, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(core): Incremental indexing via rebuild strategy

1 participant