Fix OOM issues during code indexing for large Swift files

[roomote]

This PR resolves Out of Memory (OOM) issues that occur when indexing codebases containing large Swift files.

Problem

Issue #5711 reported OOM errors during code indexing when processing Swift, Go, and C code. The issue specifically mentioned complaints about Swift language processing followed by memory issues.

Root Causes Identified

1. Large Swift files: Swift files can be memory-intensive to parse with tree-sitter
2. Unbounded memory accumulation: Batch processing accumulated code blocks without memory limits
3. No memory monitoring: No early detection of memory pressure during parsing
4. Tree-sitter memory leaks: Parser resources not properly cleaned up after processing

Solution

Memory Monitoring System

• Added MemoryMonitor utility class for tracking memory usage and pressure
• Implements singleton pattern with configurable memory threshold (85%)
• Provides memory statistics, pressure detection, and forced garbage collection

Swift-Specific Optimizations

• Reduced file size limit for Swift files from 1MB to 512KB
• Added Swift-specific memory monitoring during parsing
• Implemented forced garbage collection after Swift file processing

Parser Optimizations

• Added memory pressure checks before parsing large files
• Limited node processing to prevent excessive memory usage (max 1000 nodes)
• Implemented proper tree-sitter resource cleanup with try/finally blocks
• Added periodic memory monitoring during node processing

Scanner Improvements

• Added memory monitoring during directory scanning
• Implemented early batch processing when memory pressure is detected
• Added file-specific size limits based on file type
• Enhanced error logging with memory usage information

Testing

• All existing tests pass (367/367)
• Added comprehensive test suite for MemoryMonitor utility
• Verified memory optimizations work correctly
• No breaking changes to existing functionality

Files Changed

• src/services/code-index/constants/index.ts - Added memory-related constants
• src/services/code-index/utils/memoryMonitor.ts - New memory monitoring utility
• src/services/code-index/processors/parser.ts - Memory optimizations and Swift handling
• src/services/code-index/processors/scanner.ts - Memory monitoring during scanning
• src/services/code-index/utils/__tests__/memoryMonitor.spec.ts - Test coverage

Fixes #5711

---

Important

Fixes OOM issues during code indexing for large Swift files by implementing memory monitoring and optimizations.

• Behavior:
  • Resolves OOM issues during code indexing for large Swift files by implementing memory monitoring and optimizations.
  • Reduces Swift file size limit from 1MB to 512KB in parser.ts and scanner.ts.
  • Adds memory pressure checks and forced garbage collection in parser.ts and scanner.ts.
• Memory Monitoring:
  • Introduces MemoryMonitor class in memoryMonitor.ts for tracking memory usage and pressure.
  • Implements singleton pattern with a memory threshold of 85%.
  • Provides methods for memory statistics, pressure detection, and garbage collection.
• Testing:
  • Adds memoryMonitor.spec.ts with comprehensive tests for MemoryMonitor functionality.
  • Ensures all existing tests pass (367/367).
• Misc:
  • Adds memory-related constants to constants/index.ts.

^{This description was created by for 7e3ad68. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

The test using setTimeout to simulate throttle reset for checkAndCleanup() introduces asynchronous behavior. Consider using async/await or fake timers to reliably wait for the timeout callback.

[adamhill]

@roomote-agent Why not just truncate the Swift file? It's better to get some of the file indexed rather than of none of it.