Comparisons

.vscode/extensions.json

@@ -1,8 +1,7 @@
 {
   "recommendations": [
     "dbaeumer.vscode-eslint",
-    "esbenp.prettier-vscode",
     "github.vscode-github-actions",
-    "vitest.explorer"
+    "esbenp.prettier-vscode"
   ]
 }

.vscode/settings.json

@@ -1,17 +1,4 @@
 {
-  "eslint.probe": [
-    "javascript",
-    "javascriptreact",
-    "typescript",
-    "typescriptreact",
-    "markdown",
-    "html",
-    "json"
-  ],
-
-  "editor.formatOnSave": true,
-  "editor.formatOnPaste": true,
-
   "cSpell.words": [
     "agentic",
     "agentify",
@@ -60,6 +47,18 @@
     "vinxi"
   ],
 
+  "eslint.probe": [
+    "javascript",
+    "javascriptreact",
+    "typescript",
+    "typescriptreact",
+    "markdown",
+    "html",
+    "json"
+  ],
+
+  "editor.formatOnSave": true,
+  "editor.formatOnPaste": true,
   "editor.rulers": [160],
 
   // Workbench
@@ -74,16 +73,5 @@
   "typescript.preferences.importModuleSpecifier": "relative",
 
   "editor.defaultFormatter": "esbenp.prettier-vscode",
-  "editor.codeActionsOnSave": ["source.formatDocument", "source.fixAll.eslint"],
-  "files.exclude": {
-    "**/.git": true,
-    "**/.nx": true
-  },
-
-  "css.validate": false,
-  "tailwindCSS.includeLanguages": {
-    "typescript": "javascript",
-    "typescriptreact": "javascript"
-  },
-  "tailwindCSS.emmetCompletions": true
+  "editor.codeActionsOnSave": ["source.formatDocument", "source.fixAll.eslint"]
 }

docs/LargeCodeBase_Plan.md

@@ -0,0 +1,243 @@
+# Handling Large Codebases in MyCoder: Research and Recommendations
+
+## Executive Summary
+
+This document presents research findings on how leading AI coding tools handle large codebases and provides strategic recommendations for enhancing MyCoder's performance with large projects. The focus is on understanding indexing and context management approaches used by Claude Code and Aider, and applying these insights to improve MyCoder's architecture.
+
+## Research Findings
+
+### Claude Code (Anthropic)
+
+While detailed technical documentation on Claude Code's internal architecture is limited in public sources, we can infer several approaches from Anthropic's general AI architecture and Claude Code's capabilities:
+
+1. **Chunking and Retrieval Augmentation**:
+   - Claude Code likely employs retrieval-augmented generation (RAG) to handle large codebases
+   - Files are likely chunked into manageable segments with semantic understanding
+   - Relevant code chunks are retrieved based on query relevance
+
+2. **Hierarchical Code Understanding**:
+   - Builds a hierarchical representation of code (project → modules → files → functions)
+   - Maintains a graph of relationships between code components
+   - Prioritizes context based on relevance to the current task
+
+3. **Incremental Context Management**:
+   - Dynamically adjusts the context window to include only relevant code
+   - Maintains a "working memory" of recently accessed or modified files
+   - Uses sliding context windows to process large files sequentially
+
+4. **Intelligent Caching**:
+   - Caches parsed code structures and embeddings to avoid repeated processing
+   - Prioritizes frequently accessed or modified files in the cache
+   - Implements a cache eviction strategy based on recency and relevance
+
+### Aider
+
+Aider's approach to handling large codebases can be inferred from its open-source codebase and documentation:
+
+1. **Git Integration**:
+   - Leverages Git to track file changes and understand repository structure
+   - Uses Git history to prioritize recently modified files
+   - Employs Git's diff capabilities to minimize context needed for changes
+
+2. **Selective File Context**:
+   - Only includes relevant files in the context rather than the entire codebase
+   - Uses heuristics to identify related files based on imports, references, and naming patterns
+   - Implements a "map-reduce" approach where it first analyzes the codebase structure, then selectively processes relevant files
+
+3. **Prompt Engineering and Chunking**:
+   - Designs prompts that can work with limited context by focusing on specific tasks
+   - Chunks large files and processes them incrementally
+   - Uses summarization to compress information about non-focal code parts
+
+4. **Caching Mechanisms**:
+   - Implements token usage optimization through caching
+   - Avoids redundant LLM calls for unchanged content
+   - Maintains a local database of file content and embeddings
+
+## Recommendations for MyCoder
+
+Based on the research findings, we recommend the following enhancements to MyCoder for better handling of large codebases:
+
+### 1. Implement a Multi-Level Indexing System
+
+```
+┌───────────────────┐
+│ Project Metadata  │
+├───────────────────┤
+│ - Structure       │
+│ - Dependencies    │
+│ - Config Files    │
+└───────┬───────────┘
+        │
+        ▼
+┌───────────────────┐         ┌───────────────────┐
+│  File Index       │         │ Symbol Database   │
+├───────────────────┤         ├───────────────────┤
+│ - Path            │◄────────┤ - Functions       │
+│ - Language        │         │ - Classes         │
+│ - Modified Date   │         │ - Variables       │
+│ - Size            │         │ - Imports/Exports │
+└───────┬───────────┘         └───────────────────┘
+        │
+        ▼
+┌───────────────────┐
+│  Semantic Index   │
+├───────────────────┤
+│ - Code Embeddings │
+│ - Doc Embeddings  │
+│ - Relationships   │
+└───────────────────┘
+```
+
+**Implementation Details:**
+- Create a lightweight indexer that runs during project initialization
+- Generate embeddings for code files, focusing on API definitions, function signatures, and documentation
+- Build a graph of relationships between files based on imports/exports and references
+- Store indexes in a persistent local database for quick loading in future sessions
+
+### 2. Develop a Smart Context Management System
+
+```
+┌─────────────────────────┐
+│ Context Manager         │
+├─────────────────────────┤
+│ ┌─────────────────────┐ │
+│ │ Working Set         │ │
+│ │ (Currently relevant │ │
+│ │ files and symbols)  │ │
+│ └─────────────────────┘ │
+│                         │
+│ ┌─────────────────────┐ │
+│ │ Relevance Scoring   │ │
+│ │ Algorithm           │ │
+│ └─────────────────────┘ │
+│                         │
+│ ┌─────────────────────┐ │
+│ │ Context Window      │ │
+│ │ Optimization        │ │
+│ └─────────────────────┘ │
+└─────────────────────────┘
+```
+
+**Implementation Details:**
+- Develop a working set manager that tracks currently relevant files
+- Implement a relevance scoring algorithm that considers:
+  - Semantic similarity to the current task
+  - Recency of access or modification
+  - Dependency relationships
+  - User attention (files explicitly mentioned)
+- Optimize context window usage by:
+  - Including full content for directly relevant files
+  - Including only signatures and documentation for related files
+  - Summarizing distant but potentially relevant code
+  - Dynamically adjusting the detail level based on available context space
+
+### 3. Implement Chunking and Progressive Loading
+
+```
+┌─────────────────────────┐
+│ Chunking Strategy       │
+├─────────────────────────┤
+│ 1. Semantic Boundaries  │
+│    (Classes/Functions)  │
+│ 2. Size-based Chunks    │
+│    with Overlap         │
+│ 3. Progressive Detail   │
+│    Loading              │
+└─────────────────────────┘
+```
+
+**Implementation Details:**
+- Chunk files at meaningful boundaries (functions, classes, modules)
+- Implement overlapping chunks to maintain context across boundaries
+- Develop a progressive loading strategy:
+  - Start with high-level project structure and relevant file summaries
+  - Load detailed chunks as needed based on the task
+  - Implement a sliding context window for processing large files
+
+### 4. Create an Intelligent Caching System
+
+```
+┌─────────────────────────┐
+│ Caching System          │
+├─────────────────────────┤
+│ ┌─────────────────────┐ │
+│ │ Token Cache         │ │
+│ │ (Avoid repeated     │ │
+│ │ tokenization)       │ │
+│ └─────────────────────┘ │
+│                         │
+│ ┌─────────────────────┐ │
+│ │ Embedding Cache     │ │
+│ │ (Store vector       │ │
+│ │ representations)    │ │
+│ └─────────────────────┘ │
+│                         │
+│ ┌─────────────────────┐ │
+│ │ Prompt Template     │ │
+│ │ Cache               │ │
+│ └─────────────────────┘ │
+└─────────────────────────┘
+```
+
+**Implementation Details:**
+- Implement a multi-level caching system:
+  - Token cache: Store tokenized representations of files to avoid re-tokenization
+  - Embedding cache: Store vector embeddings for semantic search
+  - Prompt template cache: Cache commonly used prompt templates
+- Develop an efficient cache invalidation strategy based on file modifications
+- Use persistent storage for caches to maintain performance across sessions
+
+### 5. Enhance Sub-Agent Coordination for Parallel Processing
+
+```
+┌─────────────────────────┐
+│ Sub-Agent Coordinator   │
+├─────────────────────────┤
+│ ┌─────────────────────┐ │
+│ │ Task Decomposition  │ │
+│ └─────────────────────┘ │
+│                         │
+│ ┌─────────────────────┐ │
+│ │ Context Distribution│ │
+│ └─────────────────────┘ │
+│                         │
+│ ┌─────────────────────┐ │
+│ │ Result Integration  │ │
+│ └─────────────────────┘ │
+└─────────────────────────┘
+```
+
+**Implementation Details:**
+- Improve task decomposition to identify parallelizable sub-tasks
+- Implement smart context distribution to sub-agents:
+  - Provide each sub-agent with only the context it needs
+  - Share common context like project structure across all sub-agents
+  - Use a shared index to avoid duplicating large context elements
+- Develop better coordination mechanisms for sub-agents:
+  - Implement a message-passing system for inter-agent communication
+  - Create a shared memory space for efficient information exchange
+  - Design a result integration system to combine outputs from multiple sub-agents
+
+## Implementation Roadmap
+
+### Phase 1: Foundation (1-2 months)
+- Develop the basic indexing system for project structure and file metadata
+- Implement a simple relevance-based context selection mechanism
+- Create a basic chunking strategy for large files
+
+### Phase 2: Advanced Features (2-3 months)
+- Implement the semantic indexing system with code embeddings
+- Develop the full context management system with working sets
+- Create the multi-level caching system
+
+### Phase 3: Optimization and Integration (1-2 months)
+- Enhance sub-agent coordination for parallel processing
+- Optimize performance with better caching and context management
+- Integrate all components into a cohesive system
+
+## Conclusion
+
+By implementing these recommendations, MyCoder can significantly improve its performance with large codebases. The multi-level indexing system will provide a comprehensive understanding of the codebase structure, while the smart context management system will ensure that the most relevant code is included in the context window. The chunking and progressive loading strategy will enable handling of files that exceed the context window size, and the intelligent caching system will optimize token usage and improve response times. Finally, enhanced sub-agent coordination will enable efficient parallel processing of large codebases.
+
+These enhancements will position MyCoder as a leading tool for AI-assisted coding, capable of handling projects of any size with intelligent context management and efficient resource utilization.