perf: Optimize sync/indexing for 43% faster performance (#352)

Signed-off-by: phernandez <[email protected]>
Co-authored-by: Claude <[email protected]>

Author: phernandez

.github/workflows/test.yml

@@ -60,10 +60,6 @@ jobs:
         run: |
           just typecheck
 
-      - name: Run type checks
-        run: |
-          just typecheck
-
       - name: Run linting
         run: |
           just lint

BUGFIX-project-delete.md

@@ -14,22 +14,34 @@ See the [README.md](README.md) file for a project overview.
 
 ### Build and Test Commands
 
-- Install: `make install` or `pip install -e ".[dev]"`
-- Run tests: `uv run pytest -p pytest_mock -v` or `make test`
+- Install: `just install` or `pip install -e ".[dev]"`
+- Run all tests (with coverage): `just test` - Runs both unit and integration tests with unified coverage
+- Run unit tests only: `just test-unit` - Fast, no coverage
+- Run integration tests only: `just test-int` - Fast, no coverage
+- Generate HTML coverage: `just coverage` - Opens in browser
 - Single test: `pytest tests/path/to/test_file.py::test_function_name`
-- Lint: `make lint` or `ruff check . --fix`
-- Type check: `make type-check` or `uv run pyright`
-- Format: `make format` or `uv run ruff format .`
-- Run all code checks: `make check` (runs lint, format, type-check, test)
-- Create db migration: `make migration m="Your migration message"`
-- Run development MCP Inspector: `make run-inspector`
+- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
+- Lint: `just lint` or `ruff check . --fix`
+- Type check: `just typecheck` or `uv run pyright`
+- Format: `just format` or `uv run ruff format .`
+- Run all code checks: `just check` (runs lint, format, typecheck, test)
+- Create db migration: `just migration "Your migration message"`
+- Run development MCP Inspector: `just run-inspector`
 
-**Note:** Project supports Python 3.10+
+**Note:** Project requires Python 3.12+ (uses type parameter syntax and `type` aliases introduced in 3.12)
+
+### Test Structure
+
+- `tests/` - Unit tests for individual components (mocked, fast)
+- `test-int/` - Integration tests for real-world scenarios (no mocks, realistic)
+- Both directories are covered by unified coverage reporting
+- Benchmark tests in `test-int/` are marked with `@pytest.mark.benchmark`
+- Slow tests are marked with `@pytest.mark.slow`
 
 ### Code Style Guidelines
 
 - Line length: 100 characters max
-- Python 3.12+ with full type annotations
+- Python 3.12+ with full type annotations (uses type parameters and type aliases)
 - Format with ruff (consistent styling)
 - Import order: standard lib, third-party, local imports
 - Naming: snake_case for functions/variables, PascalCase for classes
@@ -63,9 +75,11 @@ See the [README.md](README.md) file for a project overview.
 - Schema changes require Alembic migrations
 - SQLite is used for indexing and full text search, files are source of truth
 - Testing uses pytest with asyncio support (strict mode)
+- Unit tests (`tests/`) use mocks when necessary; integration tests (`test-int/`) use real implementations
 - Test database uses in-memory SQLite
-- Avoid creating mocks in tests in most circumstances.
-- Each test runs in a standalone environment with in memory SQLite and tmp_file directory
+- Each test runs in a standalone environment with in-memory SQLite and tmp_file directory
+- Performance benchmarks are in `test-int/test_sync_performance_benchmark.py`
+- Use pytest markers: `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
 
 ### Async Client Pattern (Important!)
 

CLAUDE.md

@@ -34,11 +34,18 @@ project and how to get started as a developer.
 
 4. **Run the Tests**:
    ```bash
-   # Run all tests
+   # Run all tests with unified coverage (unit + integration)
    just test
-   # or
-   uv run pytest -p pytest_mock -v
-   
+
+   # Run unit tests only (fast, no coverage)
+   just test-unit
+
+   # Run integration tests only (fast, no coverage)
+   just test-int
+
+   # Generate HTML coverage report
+   just coverage
+
    # Run a specific test
    pytest tests/path/to/test_file.py::test_function_name
    ```
@@ -134,7 +141,7 @@ agreement to the DCO.
 
 ## Code Style Guidelines
 
-- **Python Version**: Python 3.12+ with full type annotations
+- **Python Version**: Python 3.12+ with full type annotations (3.12+ required for type parameter syntax)
 - **Line Length**: 100 characters maximum
 - **Formatting**: Use ruff for consistent styling
 - **Import Order**: Standard lib, third-party, local imports
@@ -144,12 +151,78 @@ agreement to the DCO.
 
 ## Testing Guidelines
 
-- **Coverage Target**: We aim for 100% test coverage for all code
+### Test Structure
+
+Basic Memory uses two test directories with unified coverage reporting:
+
+- **`tests/`**: Unit tests that test individual components in isolation
+  - Fast execution with extensive mocking
+  - Test individual functions, classes, and modules
+  - Run with: `just test-unit` (no coverage, fast)
+
+- **`test-int/`**: Integration tests that test real-world scenarios
+  - Test full workflows with real database and file operations
+  - Include performance benchmarks
+  - More realistic but slower than unit tests
+  - Run with: `just test-int` (no coverage, fast)
+
+### Running Tests
+
+```bash
+# Run all tests with unified coverage report
+just test
+
+# Run only unit tests (fast iteration)
+just test-unit
+
+# Run only integration tests
+just test-int
+
+# Generate HTML coverage report
+just coverage
+
+# Run specific test
+pytest tests/path/to/test_file.py::test_function_name
+
+# Run tests excluding benchmarks
+pytest -m "not benchmark"
+
+# Run only benchmark tests
+pytest -m benchmark test-int/test_sync_performance_benchmark.py
+```
+
+### Performance Benchmarks
+
+The `test-int/test_sync_performance_benchmark.py` file contains performance benchmarks that measure sync and indexing speed:
+
+- `test_benchmark_sync_100_files` - Small repository performance
+- `test_benchmark_sync_500_files` - Medium repository performance
+- `test_benchmark_sync_1000_files` - Large repository performance (marked slow)
+- `test_benchmark_resync_no_changes` - Re-sync performance baseline
+
+Run benchmarks with:
+```bash
+# Run all benchmarks (excluding slow ones)
+pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"
+
+# Run all benchmarks including slow ones
+pytest test-int/test_sync_performance_benchmark.py -v -m benchmark
+
+# Run specific benchmark
+pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v
+```
+
+See `test-int/BENCHMARKS.md` for detailed benchmark documentation.
+
+### Testing Best Practices
+
+- **Coverage Target**: We aim for high test coverage for all code
 - **Test Framework**: Use pytest for unit and integration tests
-- **Mocking**: Use pytest-mock for mocking dependencies only when necessary
+- **Mocking**: Avoid mocking in integration tests; use sparingly in unit tests
 - **Edge Cases**: Test both normal operation and edge cases
 - **Database Testing**: Use in-memory SQLite for testing database operations
 - **Fixtures**: Use async pytest fixtures for setup and teardown
+- **Markers**: Use `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
 
 ## Release Process
 

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
 [![PyPI version](https://badge.fury.io/py/basic-memory.svg)](https://badge.fury.io/py/basic-memory)
-[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
 [![Tests](https://github.com/basicmachines-co/basic-memory/workflows/Tests/badge.svg)](https://github.com/basicmachines-co/basic-memory/actions)
 [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 ![](https://badge.mcpx.dev?type=server 'MCP Server')

README.md

@@ -7,23 +7,28 @@ install:
     @echo ""
     @echo "💡 Remember to activate the virtual environment by running: source .venv/bin/activate"
 
-# Run unit tests in parallel
+# Run unit tests only (fast, no coverage)
 test-unit:
-    uv run pytest -p pytest_mock -v -n auto
+    uv run pytest -p pytest_mock -v --no-cov -n auto tests
 
-# Run integration tests in parallel
+# Run integration tests only (fast, no coverage)
 test-int:
     uv run pytest -p pytest_mock -v --no-cov -n auto test-int
 
-# Run all tests
+# Run all tests with unified coverage report
 test: test-unit test-int
 
+# Generate HTML coverage report
+coverage:
+    uv run pytest -p pytest_mock -v -n auto tests test-int --cov-report=html
+    @echo "Coverage report generated in htmlcov/index.html"
+
 # Lint and fix code (calls fix)
 lint: fix
 
 # Lint and fix code
 fix:
-    uv run ruff check --fix --unsafe-fixes src tests
+    uv run ruff check --fix --unsafe-fixes src tests test-int
 
 # Type check code
 typecheck:

justfile

@@ -3,7 +3,7 @@ name = "basic-memory"
 dynamic = ["version"]
 description = "Local-first knowledge management combining Zettelkasten with knowledge graphs"
 readme = "README.md"
-requires-python = ">=3.12.1"
+requires-python = ">=3.12"
 license = { text = "AGPL-3.0-or-later" }
 authors = [
     { name = "Basic Machines", email = "[email protected]" }
@@ -54,16 +54,20 @@ build-backend = "hatchling.build"
 [tool.pytest.ini_options]
 pythonpath = ["src", "tests"]
 addopts = "--cov=basic_memory --cov-report term-missing"
-testpaths = ["tests"]
+testpaths = ["tests", "test-int"]
 asyncio_mode = "strict"
 asyncio_default_fixture_loop_scope = "function"
+markers = [
+    "benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
+    "slow: Slow-running tests (deselect with '-m \"not slow\"')",
+]
 
 [tool.ruff]
 line-length = 100
 target-version = "py312"
 
-[tool.uv]
-dev-dependencies = [
+[dependency-groups]
+dev = [
     "gevent>=24.11.1",
     "icecream>=2.1.3",
     "pytest>=8.3.4",

pyproject.toml

@@ -11,9 +11,9 @@
     # Hidden files (files starting with dot)
     ".*",
     # Basic Memory internal files
-    "memory.db",
-    "memory.db-shm",
-    "memory.db-wal",
+    "*.db",
+    "*.db-shm",
+    "*.db-wal",
     "config.json",
     # Version control
     ".git",
@@ -84,10 +84,10 @@ def create_default_bmignore() -> None:
 # Hidden files (files starting with dot)
 .*
 
-# Basic Memory internal files
-memory.db
-memory.db-shm
-memory.db-wal
+# Basic Memory internal files (includes test databases)
+*.db
+*.db-shm
+*.db-wal
 config.json
 
 # Version control