Refactor indexer into a package and modularize core responsibilities for maintainability and AI-first architecture

[bashandbone]

Problem Statement

codeweaver.engine.indexer.indexer.py has grown beyond 2000+ LOC and encapsulates a wide range of responsibilities as the main orchestrator for background indexing in the codeweaver engine (currently on branch 003-our-aim-to). Its scope now includes advanced deduplication, nuanced state management, and sophisticated resolution for incomplete data/records due to recent feature expansions. While its complexity is partly justified by its central orchestration role, the resulting monolithic structure poses significant challenges for maintainability, cognitive load, and for facilitating AI-first development principles—particularly for coding agents and future contributors.

Desired Outcome

• Refactor codeweaver.engine.indexer.indexer.py into a dedicated package, distributing its core responsibilities into focused modules/classes/subpackages.
• Adopt clear separation of concerns (e.g., orchestration, state management, deduplication, incomplete data resolution) so each can be reasoned about and tested in isolation
• Design the structure for extensibility and discoverability, prioritizing ease-of-use for coding agents and new contributors
• Revisit existing hierarchical/stateful logic for opportunities for simplification and documentation
• Ensure all current functionality is preserved and verified by existing and new tests
• Add architectural documentation to guide future refactoring and integration efforts
• Prepare for future integration with pydantic-graph (see existing Agentic Phase 4 issue), anticipating further delegation of orchestration logic

Additional Context

• Branch: 003-our-aim-to (these files to appear in main on alpha release 1)
• Pydantic-graph integration planned for Alpha Feature Release 3
• Strong focus on AI-first principles: code should be easily navigated and reasoned about by coding agents

---

Labels: refactor, architecture, indexer, agents

Related Issues:

• [Agentic Phase 4] Pipeline orchestration with pydantic-graph ([Agentic Phase 4] Pipeline orchestration with pydantic-graph #127)

[github-actions]

Claude finished @bashandbone's task —— View job

---

Analyzing Issue: Refactor Indexer Package

Task List

• Review issue and gather context
• Analyze current indexer structure on 003-our-aim-to branch
• Review related issues and documentation
• Provide architectural analysis and recommendations
• Suggest labels and identify potential implementation approach

---

Issue Summary

This issue requests refactoring the codeweaver.engine.indexer.indexer.py file (currently 2,526 lines) into a modular package structure. The file has grown to encompass multiple responsibilities and needs to be broken down for better maintainability and AI-first development principles.

---

Current Indexer Analysis

Current Package Structure (003-our-aim-to branch)

src/codeweaver/engine/indexer/
├── __init__.py
├── checkpoint.py      # CheckpointManager, IndexingCheckpoint
├── indexer.py         # Main Indexer class (2526 LOC)
├── manifest.py        # FileManifestManager, IndexFileManifest
└── progress.py        # Progress tracking

Key Components in indexer.py

Classes & Types (5):

• IndexingError (TypedDict) - Structured error tracking
• ProgressCallback (Protocol) - Progress reporting interface
• UserProviderSelectionDict (TypedDict) - Provider configuration
• IndexingStats - Statistics tracking dataclass
• Indexer (BasedModel) - Main orchestrator class

Methods in Indexer Class (40+ methods):

Category	Methods	Lines (approx)
Provider Management	_initialize_providers_async, _get_current_embedding_models, _get_embedding_instance, _get_vector_store_instance	~150
File Discovery	_discover_files_to_index, _discover_files_for_batch	~120
Chunking	_chunk_discovered_files	~50
Embedding	_embed_chunks, _embed_chunks_in_batches, _retrieve_embedded_chunks	~200
Storage/Indexing	_index_file, _index_chunks_to_store, _delete_file, _cleanup_deleted_files	~200
Pipeline Orchestration	prime_index, index, _index_files_batch, _perform_batch_indexing_async	~400
Phase Methods	_phase_discovery, _phase_chunking, _phase_embed_and_index	~150
Checkpoint/State	save_checkpoint, load_checkpoint, _try_restore_from_checkpoint, _should_checkpoint	~200
Manifest Management	_load_file_manifest, _save_file_manifest, _update_manifest_for_batch	~100
Validation/Resolution	validate_manifest_with_vector_store, add_missing_embeddings_to_existing_chunks	~200
Signal Handling	_register_signal_handlers, _cleanup_signal_handlers	~50
Factory/Initialization	__init__, from_settings, from_settings_async	~250

---

Recommended Refactoring Structure

Based on the constitutional principles (especially V. Simplicity Through Architecture) and the goal of preparing for pydantic-graph integration (#127), here's a proposed modular architecture:

src/codeweaver/engine/indexer/
├── __init__.py              # Public API exports
├── base.py                  # Base protocols and types (IndexingError, ProgressCallback, etc.)
├── stats.py                 # IndexingStats class
├── checkpoint.py            # CheckpointManager (existing)
├── manifest.py              # FileManifestManager (existing)
├── progress.py              # Progress tracking (existing)
├── providers/
│   ├── __init__.py
│   ├── manager.py           # ProviderManager class - initializes/manages all providers
│   └── failover.py          # Failover logic (moved from engine/failover.py if needed)
├── pipeline/
│   ├── __init__.py
│   ├── discovery.py         # FileDiscoveryPhase - discover_files_to_index logic
│   ├── chunking.py          # ChunkingPhase - chunk_discovered_files logic
│   ├── embedding.py         # EmbeddingPhase - embed_chunks_in_batches logic
│   └── storage.py           # StoragePhase - index_chunks_to_store logic
├── operations/
│   ├── __init__.py
│   ├── batch.py             # BatchProcessor - batch indexing logic
│   └── incremental.py       # IncrementalProcessor - file change handling
├── validation/
│   ├── __init__.py
│   └── resolver.py          # ManifestValidator - validate_manifest_with_vector_store, add_missing_embeddings
└── indexer.py               # Slim Indexer class - orchestrates all components

Benefits of This Structure

1. Separation of Concerns: Each module has a single responsibility
2. AI-First Navigation: Clear, discoverable paths for coding agents
3. Testability: Each component can be tested in isolation
4. pydantic-graph Ready: Pipeline phases map cleanly to graph nodes (see [Agentic Phase 4] Pipeline orchestration with pydantic-graph #127)
5. Constitutional Compliance: Flat structure, obvious purpose per module

---

Related Issues & Context

Issue	Title	Relevance
#127	Pipeline orchestration with pydantic-graph	Future integration - pipeline phases should become graph nodes
#116	Replace registries with DI system	Provider management may be affected by DI changes
#121	Cleanup and deprecation prep	Coordinate breaking changes

The pydantic-graph integration (#127) plans to create graph nodes for:

• Search nodes
• Agent reasoning nodes
• Data provider nodes
• Analysis nodes
• Synthesis nodes

The proposed pipeline structure (discovery.py, chunking.py, embedding.py, storage.py) aligns well with this - each phase can become a graph node.

---

Implementation Difficulty Assessment

Overall: Medium-High

Factor	Rating	Notes
Scope	High	2526 LOC, 40+ methods to reorganize
Risk	Medium	Well-tested functionality, but complex interdependencies
Complexity	Medium	Clear separation boundaries, but state management is intricate
Testing Effort	High	Need to verify all existing functionality preserved
Documentation	Medium	Architecture docs needed for new structure

Key Challenges

1. State Management: The Indexer class maintains significant state across operations (checkpoints, manifests, provider instances)
2. Private Attribute Dependencies: Many PrivateAttr fields that need careful handling
3. Async Complexity: Mix of sync and async methods with complex flow
4. Incremental Indexing Logic: Sophisticated logic for change detection and resolution

Estimated Effort

• Planning & Design: 1-2 days
• Core Refactoring: 3-5 days
• Test Updates: 2-3 days
• Documentation: 1 day
• Total: ~1-2 weeks

---

Suggested Labels

Based on available repository labels, these should be applied:

• refactor - Primary purpose of the issue
• architecture - Architectural restructuring
• indexer - Component being modified
• enhancement - Improves existing functionality

(Note: I cannot assign labels directly, but these are the recommended ones)

---

Recommendations

1. Phase the Implementation: Break into smaller PRs:

   • PR 1: Extract types/protocols to base.py and stats.py
   • PR 2: Extract provider management to providers/
   • PR 3: Extract pipeline phases to pipeline/
   • PR 4: Extract operations to operations/
   • PR 5: Extract validation to validation/
   • PR 6: Slim down main indexer.py orchestrator

2. Add Integration Tests First: Before refactoring, ensure comprehensive tests exist for:

   • Full indexing pipeline (prime_index)
   • Incremental indexing
   • Checkpoint resume
   • Manifest validation

3. Document Component Contracts: Each extracted module should have clear docstrings defining:

   • Inputs/outputs
   • Side effects
   • Dependencies

4. Consider Dependency Injection: Align with [EPIC] DI + Monorepo: Unified architecture transformation #116 (DI epic) - new components should be DI-ready

---

No Duplicate Issues Found

This issue is unique - no existing issues cover the same refactoring scope. The related issues (#116, #121, #127) are complementary but distinct.

---

Branch & Job Info

• Branch: claude/issue-148-20251123-1625