[Monorepo Phase 3] Organize into clean package structure (9 packages)

[bashandbone]

Phase 3: Monorepo Structure Organization

Parent Epic: #116
Depends On: #118 (DI Integration)
Target: Week 3 (5-7 days)
Risk Level: Low

Organize CodeWeaver into clean monorepo structure with 9 independently-buildable packages, now trivial thanks to DI breaking circular dependencies.

Goals

• Organize code into packages/ with uv workspace
• All packages build independently
• Remaining violations: ~30-40 (down from 164)
• Clean dependency graph
• Documentation and migration guides
• Zero functional changes

Why This Phase is Now Easy

Thanks to Phase 2 (DI Integration):

• ✅ Circular dependencies broken (75% eliminated)
• ✅ Services don't import across packages
• ✅ Clean dependency flow established
• ✅ Just need to organize files into packages

Original estimate: 3-4 weeks to refactor dependencies
New reality: 5-7 days to organize structure (dependencies already fixed!)

Target Monorepo Structure

packages/
  codeweaver-core/
    - Core types, exceptions
    - DI infrastructure
    - search_types (moved in Phase 1)
    - No external dependencies (except stdlib)

  codeweaver-tokenizers/  ✅ (Extracted in Phase 1)
    - Tokenizer implementations
    - Tree-sitter integrations
    
  codeweaver-daemon/  ✅ (Extracted in Phase 1)
    - Background daemon logic
    - Process management

  codeweaver-utils/
    - Common utilities
    - git, logging, procs
    - Depends: core

  codeweaver-semantic/
    - Semantic chunking
    - AST analysis
    - Depends: core, utils, tokenizers

  codeweaver-telemetry/
    - Telemetry client (DI-enabled)
    - Analytics
    - Depends: core

  codeweaver-providers/
    - All provider implementations
    - Embedding, vector store, reranking
    - Provider factories (DI)
    - Depends: core, telemetry

  codeweaver-engine/
    - Indexer, search services
    - Config, registry (simplified)
    - Depends: core, utils, semantic, providers

  codeweaver/
    - CLI, server, MCP
    - agent_api orchestration
    - Depends: engine, all other packages

Implementation Checklist

Part A: Package Structure Setup (Days 1-2)

Create package directories:

• Create packages/ directory structure
• Create pyproject.toml for each package
• Set up uv workspace configuration
• Define inter-package dependencies

uv workspace configuration:

# Root pyproject.toml
[tool.uv.workspace]
members = [
    "packages/codeweaver-core",
    "packages/codeweaver-tokenizers",
    "packages/codeweaver-daemon",
    "packages/codeweaver-utils",
    "packages/codeweaver-semantic",
    "packages/codeweaver-telemetry",
    "packages/codeweaver-providers",
    "packages/codeweaver-engine",
    "packages/codeweaver",
]

Part B: Move Code to Packages (Days 3-4)

Priority 1: Foundation packages (already extracted)

• codeweaver-tokenizers ✅ (Phase 1)
• codeweaver-daemon ✅ (Phase 1)
• codeweaver-core
  • Move DI infrastructure
  • Include search_types
  • Core exceptions

Priority 2: Utility packages

• codeweaver-utils

  • Move common/utils (except registry)
  • Update imports
  • Validate independence

• codeweaver-telemetry

  • Move common/telemetry
  • DI-enabled client
  • Update imports

Priority 3: Semantic and providers

• codeweaver-semantic

  • Move semantic package
  • Update imports
  • Validate against utils

• codeweaver-providers

  • Move all provider implementations
  • Include provider factories
  • Update imports

Priority 4: Engine and app

• codeweaver-engine

  • Move engine, config
  • Simplified registry (thin layer)
  • Services using DI

• codeweaver (main package)

  • CLI, server, MCP
  • agent_api
  • Orchestration layer

Part C: Build System (Days 5-6)

Package build configuration:

• Configure build backend for each package
• Set version management strategy
• Define dependency ranges
• Set up development dependencies

Example package pyproject.toml:

[project]
name = "codeweaver-providers"
version = "0.2.0-alpha"
dependencies = [
    "codeweaver-core >=0.2.0",
    "codeweaver-telemetry >=0.2.0",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Build validation:

• Build each package independently
• Verify import paths
• Test inter-package dependencies
• Validate circular dependency elimination

Part D: Testing & Validation (Day 7)

Dependency validation:

• Run: python scripts/validate_proposed_structure.py
• Target: < 50 violations (down from 164)
• Verify no circular dependencies between packages
• Check dependency graph is acyclic

Build testing:

• Build all packages in dependency order
• Run tests for each package
• Integration test full system
• Performance validation

Expected results:

# Should succeed for all packages
cd packages/codeweaver-core && uv build
cd packages/codeweaver-utils && uv build
cd packages/codeweaver-providers && uv build
# ... etc

Package Dependency Graph

Clean dependency flow (acyclic):

codeweaver-core (foundation)
  ↑
  ├── codeweaver-tokenizers
  ├── codeweaver-daemon
  ├── codeweaver-utils
  ├── codeweaver-telemetry
  ↑
  ├── codeweaver-semantic (depends on: core, utils, tokenizers)
  ├── codeweaver-providers (depends on: core, telemetry)
  ↑
  ├── codeweaver-engine (depends on: core, utils, semantic, providers)
  ↑
  └── codeweaver (depends on: ALL)

No circular dependencies between packages!

Remaining Violations (~30-40)

What's left after DI broke 75%:

Type movements (10-15 violations)

• Some types still in wrong packages
• Easy to fix: just move files
• No logic changes needed

Utility dependencies (9 violations)

• core → utils
• Solution: Move core utilities to core package

Semantic utilities (4 violations)

• semantic → utils
• Solution: Move or minimize shared utilities

Minor couplings (10-15 violations)

• Various small import adjustments
• Lazy imports where needed
• Protocol usage for abstract dependencies

Acceptance Criteria

• All 9 packages created with pyproject.toml
• Code organized into packages
• uv workspace builds successfully
• Dependency violations < 50 (down from 164)
• Zero circular dependencies between packages
• All packages build independently
• All tests pass
• Type checking passes
• Performance within 5% of baseline
• Documentation complete

Migration Guide

Document for users:

• Import path changes
• Package installation instructions
• Development setup with workspace
• Contribution guidelines per package

Example migration:

# Before (monolith)
from codeweaver.engine.indexer import Indexer
from codeweaver.providers.embedding.fastembed import FastembedProvider

# After (monorepo)
from codeweaver_engine.indexer import Indexer
from codeweaver_providers.embedding.fastembed import FastembedProvider

Benefits After This Phase

For developers:

• ✅ Work on individual packages without full codebase
• ✅ Clear package boundaries and responsibilities
• ✅ Independent versioning possible
• ✅ Faster builds (only rebuild changed packages)

For users:

• ✅ Install only needed packages
• ✅ Lighter dependencies for specific use cases
• ✅ Clear module structure

For maintainers:

• ✅ Easier to review (package-scoped changes)
• ✅ Independent package releases
• ✅ Clear ownership boundaries

Success Metrics

Structural:

• 9 packages building independently
• < 50 dependency violations
• 0 circular dependencies between packages
• Clean dependency graph

Quality:

• All tests pass
• Type checking passes
• Documentation complete
• Migration guide available

Next Steps After This Phase

Phase 4 (#119): pydantic-ai Integration

• Add agent DI support
• Integrate data providers
• Build on clean package structure

Phase 5 (#120): Advanced Features

• Health checks, telemetry
• Plugin system
• Performance optimization

Phase 6 (#121): Cleanup

• Deprecate old patterns
• Eliminate/simplify registry
• Final documentation

Connection to Integrated Strategy

This phase implements Week 3 of the integrated strategy:

• Organize into packages → Natural structure emerges
• Build system → All packages independent
• Final validation → Proof of clean architecture

Why This is Low Risk Now

Thanks to Phases 1-2:

• ✅ Circular dependencies already broken (DI work)
• ✅ Services already don't import across boundaries
• ✅ Just moving files, not changing logic
• ✅ Can validate continuously during organization

Original risk: HIGH (dependencies blocking)
Current risk: LOW (just file organization)

Reference

• Planning: INTEGRATED_DI_MONOREPO_STRATEGY.md (Week 3)
• Visualization: DI_IMPACT_VISUALIZATION.md
• Original analysis: REFINED_MONOREPO_ANALYSIS.md
• Validation script: scripts/validate_proposed_structure.py

[github-actions]

Claude finished @bashandbone's task —— View job • claude/issue-204-20251211-1543 • Create PR ➔

---

Phase 3 Monorepo Structure Analysis

I've completed a thorough review of this Phase 3 issue and its relationship to the broader DI + Monorepo transformation epic. Here's my comprehensive analysis:

---

Issue Summary

Phase 3: Monorepo Structure Organization is part of a larger architectural transformation (#116) to reorganize CodeWeaver into a clean monorepo with 9 independently-buildable packages. This phase depends on the completion of Phase 2 (#118 - DI Integration), which eliminates 75% of circular dependencies through dependency injection.

Key Goals:

• Organize code into packages/ with uv workspace
• Reduce violations from 164 to ~30-40 (now ~75% already handled by DI)
• Create 9 independent packages
• Zero functional changes
• Clean dependency graph

Timeline: 5-7 days (Week 3 of the integrated strategy)
Risk Level: Low (thanks to DI work breaking dependencies first)

---

Related Issues & Context

Parent & Dependencies:

• [EPIC] DI + Monorepo: Unified architecture transformation #116 - [EPIC] DI + Monorepo: Unified architecture transformation (OPEN)
• [DI Phase 2] Core migration - Breaking 75% of circular dependencies #118 - [DI Phase 2] Core migration - Breaking 75% of circular dependencies (OPEN, in progress) ← BLOCKS THIS ISSUE
• [DI Phase 1] Foundation + monorepo prep (tokenizers, daemon, SearchResult) #117 - [DI Phase 1] Foundation + monorepo prep (OPEN) ← ALSO BLOCKS THIS ISSUE

Follow-up Phases:

• [DI Phase 4] pydantic-ai integration (agents + data sources) #119 - [DI Phase 4] pydantic-ai integration
• [DI Phase 5] Advanced features (health, telemetry, plugins) #120 - [DI Phase 5] Advanced features (health, telemetry, plugins)
• [DI Phase 6] Cleanup, registry simplification, and v0.4 breaking changes #121 - [DI Phase 6] Cleanup, registry simplification, breaking changes

---

Current Status Assessment

✅ What Exists:

• Clear architectural vision in CLAUDE.md
• Comprehensive DI planning documentation in plans/ directory
• Well-defined package structure proposal
• Current monolithic structure in src/codeweaver/

❌ What's Missing (Blockers):

1. Phase 1 ([DI Phase 1] Foundation + monorepo prep (tokenizers, daemon, SearchResult) #117) not completed - DI foundation needs to be implemented first
2. Phase 2 ([DI Phase 2] Core migration - Breaking 75% of circular dependencies #118) not completed - Core migration to DI needs to happen before package split
3. No packages/ directory exists yet
4. No validation script (scripts/validate_proposed_structure.py mentioned but doesn't exist)
5. Referenced documentation missing:
   • INTEGRATED_DI_MONOREPO_STRATEGY.md
   • DI_IMPACT_VISUALIZATION.md
   • REFINED_MONOREPO_ANALYSIS.md

---

Key Findings & Challenges

1. Dependency Chain Not Met

This issue explicitly states it depends on #118 (Phase 2), but Phase 2 is still OPEN and appears not yet implemented. The issue description claims "Thanks to Phase 2 (DI Integration):" but those achievements haven't been realized yet.

Status: Phase 2 checklist shows all tasks uncompleted (all [ ]).

2. Missing Tooling

The issue references a validation script (scripts/validate_proposed_structure.py) that doesn't exist. This tool would be critical for:

• Validating the current 164 violations
• Tracking progress toward the ~30-40 target
• Ensuring no circular dependencies between packages

Recommendation: This validation tool should be created as part of Phase 1 or 2, before attempting Phase 3.

3. Documentation Gap

Several key planning documents referenced in the issue are missing. However, there ARE comprehensive DI planning docs in plans/:

• dependency-injection-architecture-plan.md (31KB)
• DI_ARCHITECTURE_SUMMARY.md (5KB)
• DI_IMPLEMENTATION_CHECKLIST.md (10KB)
• DI_ARCHITECTURE_DIAGRAMS.md (21KB)

4. Phase 1 Claims vs Reality

The issue states packages were extracted in Phase 1, but these don't exist in packages/ directory yet.

---

Feasibility Assessment

Can Phase 3 Start Now? ❌ NO

Reasons:

1. Blocking dependencies incomplete: Phases 1 and 2 must complete first
2. DI infrastructure doesn't exist yet - The circular dependency elimination that makes this "trivial" hasn't happened
3. Original estimate was based on DI completion: "5-7 days" assumes DI work is done
4. Missing validation tooling to measure progress

Timeline Realism:

Actual Situation: Phases 1 and 2 need ~12-17 days before Phase 3 can begin.

Revised Estimate:

• Phase 1: 5-7 days (DI Foundation)
• Phase 2: 7-10 days (Core Migration)
• Phase 3: 5-7 days (Monorepo Structure) ← This issue
• Total from current state: 17-24 days (3-4 weeks)

The 5-7 day estimate for Phase 3 is realistic, but only after Phase 2 completes.

---

Recommendations

1. Update Issue Status

Add a prominent note at the top indicating this is blocked:

> **⚠️ BLOCKED:** This issue depends on #117 and #118 completion.  
> **Status:** Cannot start until DI infrastructure is implemented.

2. Create Missing Tooling First

Before any Phase 3 work:

• Create scripts/validate_proposed_structure.py to measure current violations
• Establish baseline: Run validation to confirm "164 violations"
• Define success criteria: What counts as a "violation"?

3. Complete Prerequisites

Ensure Phase 1 and 2 completion:

• [DI Phase 1] Foundation + monorepo prep (tokenizers, daemon, SearchResult) #117 - DI Foundation fully implemented
• [DI Phase 2] Core migration - Breaking 75% of circular dependencies #118 - Core services migrated to DI
• Validate circular dependency reduction (should see ~120-130 eliminated)
• Confirm remaining violations are ~30-40

4. Consider Splitting the Issue

Phase 3 has 4 distinct parts over 7 days - consider creating sub-issues for better tracking.

---

Constitutional Compliance

This phase aligns well with CodeWeaver's constitutional principles:

✅ Principle II (Proven Patterns): Uses standard uv workspace, following established monorepo patterns
✅ Principle III (Evidence-Based): Requires validation script and measurable metrics
✅ Principle V (Simplicity): Clear package boundaries, obvious purpose
⚠️ Concern: Issue description states work as "done" when it's actually blocked

---

Suggested Labels

Recommended labels:

• enhancement ✅ (new feature)
• architecture ✅ (structural change)
• monorepo (create if doesn't exist - this is THE monorepo implementation issue)
• blocked ← Add this (currently blocked by [DI Phase 1] Foundation + monorepo prep (tokenizers, daemon, SearchResult) #117 and [DI Phase 2] Core migration - Breaking 75% of circular dependencies #118)
• refactor (restructuring existing code)
• dependency-injection (related to DI work)

---

Difficulty & Impact Estimate

Difficulty:

• Current state (without Phases 1-2): HIGH - Would require manual dependency untangling
• After Phases 1-2 complete: LOW-MEDIUM - Mostly file organization with careful import updates

Impact:

• User-facing: NONE (zero functional changes stated)
• Developer experience: HIGH - Clean package boundaries, faster builds
• Architecture: HIGH - Enables future independent package releases
• Risk: LOW (after DI work) - Just moving files

API Changes: None stated (internal reorganization only)

Breaking Changes: Import paths will change, but manageable with migration guide

---

Implementation Risks

Even after Phase 2 completes, watch for:

1. Import path updates - Comprehensive search/replace needed
2. Test imports - Tests may have hardcoded paths
3. Build system complexity - uv workspace configuration can be tricky
4. Version management - 9 packages need coordinated versioning
5. CI/CD updates - Build pipeline needs to handle workspace
6. Documentation updates - All docs need new import paths

---

Deliverables

I've created a comprehensive analysis document for the repository:

📄 plans/PHASE_3_MONOREPO_ANALYSIS.md

• Current status and blocking dependencies
• Timeline reconciliation
• Missing tooling identification
• Risk assessment
• Implementation recommendations
• Constitutional compliance validation

This document provides detailed context for when Phase 3 can begin.

---

Conclusion

This is a well-planned and well-documented issue that correctly identifies the dependency on Phase 2 DI work. However, it's currently blocked and cannot proceed until the prerequisite phases are completed.

Next Steps:

1. ✅ Complete Phase 1 ([DI Phase 1] Foundation + monorepo prep (tokenizers, daemon, SearchResult) #117) - DI Foundation
2. ✅ Complete Phase 2 ([DI Phase 2] Core migration - Breaking 75% of circular dependencies #118) - Core Migration
3. Create validation tooling
4. Verify 75% reduction in circular dependencies
5. THEN proceed with Phase 3

Timeline from Today:

• Phases 1-2: ~12-17 days
• Phase 3: ~5-7 days
• Total: ~3-4 weeks to monorepo structure

The 5-7 day estimate for Phase 3 is realistic only after the DI work is complete.

---

| Branch: claude/issue-204-20251211-1543