Refactor toolAgent.ts into modular components

.changeset/convert-to-zod.md

@@ -0,0 +1,6 @@
+---
+'mycoder-agent': minor
+'mycoder': minor
+---
+
+Convert from JsonSchema7Type to ZodSchema for tool parameters and returns, required for Vercel AI SDK integration.

.changeset/implement-token-caching.md

@@ -0,0 +1,5 @@
+---
+'mycoder-agent': patch
+---
+
+Re-implemented token caching for Vercel AI SDK usage with Anthropic provider to reduce token consumption during repeated API calls.

.changeset/refactor-tool-agent.md

@@ -0,0 +1,5 @@
+---
+'mycoder-agent': minor
+---
+
+Refactored toolAgent.ts into modular components for improved maintainability and testability. Split into config.ts, messageUtils.ts, toolExecutor.ts, tokenTracking.ts, and types.ts modules.

CONTRIBUTING.md

@@ -80,17 +80,19 @@ This project and everyone participating in it is governed by our Code of Conduct
 5. Push to your fork and create a Pull Request
 
 6. Pre-commit Hooks:
-   
+
    We use [husky](https://typicode.github.io/husky/) and [lint-staged](https://github.com/okonet/lint-staged) to automatically run linting and formatting on staged files before each commit. This helps maintain code quality and consistency.
 
    The pre-commit hooks are configured to run:
-   - `pnpm lint`: Lints the staged files using ESLint 
+
+   - `pnpm lint`: Lints the staged files using ESLint
    - `pnpm format`: Formats the staged files using Prettier
 
    If either of these commands fails due to linting errors or formatting issues, the commit will be aborted. Please fix the reported issues and try committing again.
 
    You can also run the lint and format commands manually at any time:
-   ```bash 
+
+   ```bash
    pnpm lint        # Lint all files
    pnpm format      # Format all files
    ```

docs/LargeCodeBase_Plan.md

@@ -11,16 +11,19 @@ This document presents research findings on how leading AI coding tools handle l
 While detailed technical documentation on Claude Code's internal architecture is limited in public sources, we can infer several approaches from Anthropic's general AI architecture and Claude Code's capabilities:
 
 1. **Chunking and Retrieval Augmentation**:
+
    - Claude Code likely employs retrieval-augmented generation (RAG) to handle large codebases
    - Files are likely chunked into manageable segments with semantic understanding
    - Relevant code chunks are retrieved based on query relevance
 
 2. **Hierarchical Code Understanding**:
+
    - Builds a hierarchical representation of code (project → modules → files → functions)
    - Maintains a graph of relationships between code components
    - Prioritizes context based on relevance to the current task
 
 3. **Incremental Context Management**:
+
    - Dynamically adjusts the context window to include only relevant code
    - Maintains a "working memory" of recently accessed or modified files
    - Uses sliding context windows to process large files sequentially
@@ -35,16 +38,19 @@ While detailed technical documentation on Claude Code's internal architecture is
 Aider's approach to handling large codebases can be inferred from its open-source codebase and documentation:
 
 1. **Git Integration**:
+
    - Leverages Git to track file changes and understand repository structure
    - Uses Git history to prioritize recently modified files
    - Employs Git's diff capabilities to minimize context needed for changes
 
 2. **Selective File Context**:
+
    - Only includes relevant files in the context rather than the entire codebase
    - Uses heuristics to identify related files based on imports, references, and naming patterns
    - Implements a "map-reduce" approach where it first analyzes the codebase structure, then selectively processes relevant files
 
 3. **Prompt Engineering and Chunking**:
+
    - Designs prompts that can work with limited context by focusing on specific tasks
    - Chunks large files and processes them incrementally
    - Uses summarization to compress information about non-focal code parts
@@ -90,6 +96,7 @@ Based on the research findings, we recommend the following enhancements to MyCod
 ```
 
 **Implementation Details:**
+
 - Create a lightweight indexer that runs during project initialization
 - Generate embeddings for code files, focusing on API definitions, function signatures, and documentation
 - Build a graph of relationships between files based on imports/exports and references
@@ -120,6 +127,7 @@ Based on the research findings, we recommend the following enhancements to MyCod
 ```
 
 **Implementation Details:**
+
 - Develop a working set manager that tracks currently relevant files
 - Implement a relevance scoring algorithm that considers:
   - Semantic similarity to the current task
@@ -148,6 +156,7 @@ Based on the research findings, we recommend the following enhancements to MyCod
 ```
 
 **Implementation Details:**
+
 - Chunk files at meaningful boundaries (functions, classes, modules)
 - Implement overlapping chunks to maintain context across boundaries
 - Develop a progressive loading strategy:
@@ -181,6 +190,7 @@ Based on the research findings, we recommend the following enhancements to MyCod
 ```
 
 **Implementation Details:**
+
 - Implement a multi-level caching system:
   - Token cache: Store tokenized representations of files to avoid re-tokenization
   - Embedding cache: Store vector embeddings for semantic search
@@ -209,6 +219,7 @@ Based on the research findings, we recommend the following enhancements to MyCod
 ```
 
 **Implementation Details:**
+
 - Improve task decomposition to identify parallelizable sub-tasks
 - Implement smart context distribution to sub-agents:
   - Provide each sub-agent with only the context it needs
@@ -222,16 +233,19 @@ Based on the research findings, we recommend the following enhancements to MyCod
 ## Implementation Roadmap
 
 ### Phase 1: Foundation (1-2 months)
+
 - Develop the basic indexing system for project structure and file metadata
 - Implement a simple relevance-based context selection mechanism
 - Create a basic chunking strategy for large files
 
 ### Phase 2: Advanced Features (2-3 months)
+
 - Implement the semantic indexing system with code embeddings
 - Develop the full context management system with working sets
 - Create the multi-level caching system
 
 ### Phase 3: Optimization and Integration (1-2 months)
+
 - Enhance sub-agent coordination for parallel processing
 - Optimize performance with better caching and context management
 - Integrate all components into a cohesive system

docs/SentryIntegration.md

@@ -17,6 +17,7 @@ npm install @sentry/node --save
 ## Configuration
 
 By default, Sentry is:
+
 - Enabled in production environments
 - Disabled in development environments (unless explicitly enabled)
 - Configured to capture 100% of transactions
@@ -56,7 +57,9 @@ Sentry.init({
   tracesSampleRate: 1.0,
   environment: process.env.NODE_ENV || 'development',
   release: `mycoder@${packageVersion}`,
-  enabled: process.env.NODE_ENV !== 'development' || process.env.ENABLE_SENTRY === 'true',
+  enabled:
+    process.env.NODE_ENV !== 'development' ||
+    process.env.ENABLE_SENTRY === 'true',
 });
 
 // Capture errors
@@ -76,6 +79,7 @@ mycoder test-sentry
 ```
 
 This command will:
+
 1. Generate a test error that includes the package version
 2. Report it to Sentry.io
 3. Output the result to the console
@@ -85,6 +89,7 @@ Note: In development environments, you may need to set `ENABLE_SENTRY=true` for
 ## Privacy
 
 Error reports sent to Sentry include:
+
 - Stack traces
 - Error messages
 - Environment information

packages/agent/package.json

@@ -44,13 +44,16 @@
   "author": "Ben Houston",
   "license": "MIT",
   "dependencies": {
-    "@anthropic-ai/sdk": "^0.37",
+    "@ai-sdk/anthropic": "^1.1.13",
+    "@ai-sdk/openai": "^1.2.0",
     "@mozilla/readability": "^0.5.0",
     "@playwright/test": "^1.50.1",
     "@vitest/browser": "^3.0.5",
+    "ai": "^4.1.50",
     "chalk": "^5.4.1",
     "dotenv": "^16",
     "jsdom": "^26.0.0",
+    "ollama-ai-provider": "^1.2.0",
     "playwright": "^1.50.1",
     "uuid": "^11",
     "zod": "^3.24.2",

packages/agent/src/core/tokens.ts

@@ -1,4 +1,4 @@
-import Anthropic from '@anthropic-ai/sdk';
+//import Anthropic from '@anthropic-ai/sdk';
 
 import { LogLevel } from '../utils/logger.js';
 
@@ -34,14 +34,15 @@ export class TokenUsage {
     return usage;
   }
 
+  /*
   static fromMessage(message: Anthropic.Message) {
     const usage = new TokenUsage();
     usage.input = message.usage.input_tokens;
     usage.cacheWrites = message.usage.cache_creation_input_tokens ?? 0;
     usage.cacheReads = message.usage.cache_read_input_tokens ?? 0;
     usage.output = message.usage.output_tokens;
     return usage;
-  }
+  }*/
 
   static sum(usages: TokenUsage[]) {
     const usage = new TokenUsage();