Add performance optimizations for large repositories

[scottbaggett]

Feature Request: Performance Optimizations

Problem Statement

Large repositories with extensive history or many files can make dex slow, especially with full file extraction.

Proposed Solution

Implement performance optimizations for better scalability.

Implementation Details

Optimizations

1. Parallel Processing

   • Concurrent file reading
   • Parallel git operations
   • Multi-threaded parsing

2. Caching

   • Cache git operations
   • Store parsed ASTs
   • Remember file contents
   • Token count cache

3. Lazy Loading

   • Stream large files
   • Incremental processing
   • On-demand symbol resolution

4. Smart Limits

   • File size thresholds
   • Token count limits
   • Automatic truncation
   • Sampling strategies

CLI Options

dex --max-file-size=1MB    # Skip large files
dex --max-tokens=100000    # Token limit
dex --cache               # Enable caching
dex --parallel=4          # Worker threads

Progress Indicators

Analyzing changes... ████████░░ 80% | 1234/1543 files | ETA: 5s

Performance Metrics

## Metadata
- **Processing Time:** 2.3s
- **Files Analyzed:** 1543
- **Cache Hits:** 89%
- **Peak Memory:** 124MB

Benefits

• 10x faster on large repos
• Lower memory usage
• Better user experience
• Scalable to monorepos