diff --git a/PLAN.md b/PLAN.md index ffd1cca..6a75a88 100644 --- a/PLAN.md +++ b/PLAN.md @@ -76,122 +76,87 @@ Dev-agent provides semantic code search, codebase intelligence, and GitHub integ --- -## Current: Context Quality (v0.2) +## Completed: Context Quality (v0.3.0) ✅ -The next phase focuses on making dev-agent's context *actually useful* for LLM reasoning. +Released in v0.3.0 - dev-agent now provides *actually useful* context for LLM reasoning. ### Principle: Structured Data Over Summaries Don't generate prose—provide structured data and let the LLM synthesize. -### Priority 1: Richer Search Results +### Richer Search Results ✅ -**Problem:** Current search returns pointers, not context. LLMs need another round-trip to read files. - -**Solution:** Include code snippets and relationship hints in search results. +| Feature | Status | +|---------|--------| +| Code snippets in search results | ✅ Done | +| Import/export context | ✅ Done | +| Callers/callees hints | ✅ Done | +| Token budget management | ✅ Done | +| Progressive disclosure | ✅ Done | -| Task | Priority | Complexity | -|------|----------|------------| -| Add code snippets to search results | 🔴 High | Low | -| Include import/export context | 🔴 High | Medium | -| Show callers/callees hints | 🟡 Medium | Medium | -| Token budget management | 🟡 Medium | Low | +### Relationship Queries (`dev_refs`) ✅ -**Before:** -``` -[85%] function: handleAuth (src/auth/handler.ts:45) -``` +| Feature | Status | +|---------|--------| +| Callee extraction during indexing | ✅ Done | +| `dev_refs` MCP adapter | ✅ Done | +| Bidirectional queries (callers/callees) | ✅ Done | +| Token budget support | ✅ Done | -**After:** -``` -[85%] function: handleAuth (src/auth/handler.ts:45-67) - Snippet: export async function handleAuth(req: Request)... - Imports: ./service, ../utils/jwt - Called by: src/routes/api.ts:23 -``` +### Codebase Map (`dev_map`) ✅ -### Priority 2: Relationship Queries (`dev_refs`) - -**Problem:** "What calls this function?" is unanswerable without reading the whole codebase. - -**Solution:** New MCP tool for relationship queries. - -| Task | Priority | Complexity | -|------|----------|------------| -| Build call graph during indexing | 🔴 High | High | -| `dev_refs` adapter | 🔴 High | Medium | -| Bidirectional queries (callers/callees) | 🟡 Medium | Medium | -| Cross-file import tracking | 🟡 Medium | Medium | - -**Example:** -```typescript -// Input -{ symbol: "handleAuth", direction: "both" } - -// Output -{ - "callers": [ - { "file": "src/routes/api.ts", "line": 23, "symbol": "authMiddleware" } - ], - "callees": [ - { "file": "src/auth/service.ts", "line": 12, "symbol": "validateToken" } - ] -} -``` +| Feature | Status | +|---------|--------| +| Directory tree with component counts | ✅ Done | +| `dev_map` MCP adapter | ✅ Done | +| Configurable depth (1-5) | ✅ Done | +| Focus on specific directories | ✅ Done | +| Export signatures | ✅ Done | +| Hot paths (most referenced files) | ✅ Done | +| Smart depth (adaptive expansion) | ✅ Done | -### Priority 3: Codebase Map (`dev_map`) +### Refactored Planner → Context Assembler ✅ -**Problem:** LLMs don't know the shape of the codebase without reading everything. +| Feature | Status | +|---------|--------| +| Removed heuristic task breakdown | ✅ Done | +| Returns `ContextPackage` with raw issue + code | ✅ Done | +| Includes codebase patterns | ✅ Done | +| Related PR/issue history | ✅ Done | -**Solution:** Structured skeleton view (inspired by Aider's repo-map). +--- -| Task | Priority | Complexity | -|------|----------|------------| -| Generate codebase skeleton | 🟡 Medium | Medium | -| `dev_map` adapter | 🟡 Medium | Low | -| Configurable depth | 🟢 Low | Low | -| Focus on specific directories | 🟢 Low | Low | +## Current: Polish & Stabilize (v0.3.x) -**Example:** -``` -src/auth/ - handler.ts - ├─ handleAuth(req: Request): Promise - ├─ validateSession(token: string): boolean - service.ts - ├─ class AuthService - │ ├─ login(credentials): Promise - │ └─ logout(userId): void -``` +Focus on quality, documentation, and developer experience before adding new features. -### Priority 4: Refactor Planner → Context Assembler +### Documentation -**Problem:** Planner generates heuristic tasks that LLMs could do better with raw data. +| Task | Status | Priority | +|------|--------|----------| +| CLI reference docs | ✅ Done | 🟡 Medium | +| Configuration guide | ✅ Done | 🟡 Medium | +| Troubleshooting guide | ✅ Done | 🟡 Medium | +| Examples for new tools | ✅ Done | 🟢 Low | -**Solution:** Return structured context, not generated plans. +### Code Quality -| Task | Priority | Complexity | -|------|----------|------------| -| Remove heuristic task breakdown | 🟡 Medium | Low | -| Return raw issue + relevant code | 🟡 Medium | Low | -| Include codebase patterns | 🟢 Low | Medium | -| Add related PR/issue history | 🟢 Low | Medium | +| Task | Status | Priority | +|------|--------|----------| +| Fix lint warnings | ✅ Done | 🔴 High | +| Context assembler tests | ✅ Done | 🟡 Medium | +| Integration tests for new tools | ✅ Done | 🟢 Low | -**Before:** Generic tasks like "Design solution", "Implement", "Test" +### Issue Cleanup -**After:** Raw materials for LLM to plan with: -```json -{ - "issue": { "title": "...", "body": "...", "labels": [...] }, - "relevantCode": [{ "file": "...", "snippet": "...", "similarity": 0.85 }], - "patterns": { "middleware": "Express-style", "testing": "Vitest + __tests__/" }, - "history": [{ "pr": 38, "title": "Similar feature", "files": [...] }] -} -``` +| Task | Status | Priority | +|------|--------|----------| +| Close completed epics | ✅ Done | 🟡 Medium | +| Update stale issues | ✅ Done | 🟢 Low | --- -## Future: Extended Intelligence (v0.3+) +## Future: Extended Intelligence (v0.4+) ### Git History Context @@ -200,7 +165,7 @@ src/auth/ | Recent commits affecting file | 🟡 Medium | | Git blame integration | 🟡 Medium | | Related PRs for file/function | 🟡 Medium | -| Change frequency analysis | 🟢 Low | +| Change frequency analysis | ✅ Done (hot paths) | ### Multi-Language Support @@ -252,7 +217,7 @@ These were in the original plan but have been deprioritized or reconsidered: - **Package Manager:** pnpm 8.15.4 - **Build:** Turborepo - **Linting:** Biome -- **Testing:** Vitest (1100+ tests) +- **Testing:** Vitest (1379+ tests) - **Vector Storage:** LanceDB (embedded, no server) - **Embeddings:** @xenova/transformers (all-MiniLM-L6-v2) - **AI Integration:** MCP (Model Context Protocol) diff --git a/examples/README.md b/examples/README.md index 74bacda..33ce2fb 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,20 +1,375 @@ # Dev-Agent Examples -This directory contains example projects and usage patterns for the Dev-Agent system. +Real-world usage patterns for dev-agent MCP tools. -## Basic Usage +## Quick Start -The examples demonstrate how to: +```bash +# Install dev-agent +npm install -g dev-agent -1. Initialize the repository context provider -2. Generate vector embeddings for your codebase -3. Query for relevant context -4. Integrate with AI tools like Claude Code +# Index your repository +cd /path/to/your/project +dev index . -## Examples +# Install MCP for Cursor +dev mcp install --cursor -- **simple-context**: A basic example of initializing and using the context provider -- **claude-integration**: Example of using Dev-Agent with Claude Code -- **subagent-demo**: Demonstration of subagent capabilities for planning and exploration +# Restart Cursor - tools are now available! +``` -To run an example, navigate to its directory and follow the instructions in its README.md file. \ No newline at end of file +--- + +## Tool Examples + +### `dev_search` - Semantic Code Search + +Find code by meaning, not exact text: + +``` +# Find authentication logic +dev_search: "user authentication and login flow" + +# Find error handling patterns +dev_search: "how errors are caught and handled" + +# Find API endpoints +dev_search: "REST API route handlers" +``` + +**With options:** +``` +dev_search: + query: "database connection pooling" + limit: 5 + scoreThreshold: 0.4 + tokenBudget: 2000 +``` + +**Output includes:** +- Code snippets (up to 50 lines) +- Import statements +- File locations +- Relevance scores + +--- + +### `dev_refs` - Relationship Queries + +Understand code dependencies: + +``` +# What calls this function? +dev_refs: + name: "validateUser" + direction: "callers" + +# What does this function call? +dev_refs: + name: "processPayment" + direction: "callees" + +# Both directions +dev_refs: + name: "AuthService" + direction: "both" +``` + +**Use cases:** +- Impact analysis before refactoring +- Understanding code flow +- Finding entry points + +--- + +### `dev_map` - Codebase Overview + +Get a high-level view of the codebase: + +``` +# Basic map (depth 2) +dev_map + +# Deeper exploration +dev_map: + depth: 4 + +# Focus on specific directory +dev_map: + focus: "src/api" + depth: 3 + +# Include hot paths (most referenced files) +dev_map: + includeHotPaths: true + smartDepth: true +``` + +**Output shows:** +- Directory structure +- Component counts per directory +- Exported symbols with signatures +- Hot paths (frequently referenced files) + +--- + +### `dev_plan` - Context Assembly + +Get rich context for implementing a GitHub issue: + +``` +# Basic context +dev_plan: + issue: 42 + +# Full context package +dev_plan: + issue: 42 + includeCode: true + includeHistory: true + includePatterns: true +``` + +**Returns:** +- Issue details (title, body, labels, comments) +- Relevant code snippets from semantic search +- Codebase patterns (test conventions, etc.) +- Related issues/PRs + +--- + +### `dev_gh` - GitHub Search + +Search issues and PRs semantically: + +``` +# Search issues +dev_gh: + action: "search" + query: "authentication bugs" + +# Get specific issue +dev_gh: + action: "get" + number: 42 +``` + +**First, index GitHub:** +```bash +dev gh index +``` + +--- + +### `dev_explore` - Code Exploration + +Find patterns and similar code: + +``` +# Find patterns +dev_explore: + action: "pattern" + query: "error handling middleware" + +# Find similar code +dev_explore: + action: "similar" + path: "src/utils/retry.ts" +``` + +--- + +### `dev_status` - Repository Status + +Check indexing status: + +``` +dev_status + +# Specific section +dev_status: + section: "indexes" +``` + +--- + +### `dev_health` - Health Check + +Diagnose issues: + +``` +dev_health + +# Verbose output +dev_health: + verbose: true +``` + +--- + +## Workflow Examples + +### Starting a New Feature + +1. **Understand the codebase:** + ``` + dev_map: { depth: 3, includeHotPaths: true } + ``` + +2. **Find related code:** + ``` + dev_search: "similar feature implementation" + ``` + +3. **Check what calls the area you'll modify:** + ``` + dev_refs: { name: "TargetModule", direction: "callers" } + ``` + +### Bug Investigation + +1. **Search for the bug area:** + ``` + dev_search: "error message from bug report" + ``` + +2. **Trace the code path:** + ``` + dev_refs: { name: "suspectFunction", direction: "callees" } + ``` + +3. **Find similar issues:** + ``` + dev_gh: { action: "search", query: "similar error" } + ``` + +### Implementing a GitHub Issue + +1. **Get full context:** + ``` + dev_plan: { issue: 123 } + ``` + +2. **Explore relevant patterns:** + ``` + dev_explore: { action: "pattern", query: "feature type from issue" } + ``` + +### Code Review Prep + +1. **Understand the change area:** + ``` + dev_map: { focus: "path/to/changed/dir" } + ``` + +2. **Check impact:** + ``` + dev_refs: { name: "changedFunction", direction: "callers" } + ``` + +--- + +## Tips + +### Search Quality + +- **Use natural language** - "how users are authenticated" not "authUser function" +- **Describe behavior** - "retry logic with exponential backoff" +- **Lower threshold for exploration** - `scoreThreshold: 0.3` + +### Token Management + +- Use `tokenBudget` to control output size +- `compact` format for quick lookups +- `verbose` format for deep dives + +### Keeping Index Fresh + +```bash +# After major changes +dev index . + +# After new issues/PRs +dev gh index + +# Check health +dev_health +``` + +--- + +## CLI Examples + +```bash +# Search from command line +dev search "authentication" --limit 5 --threshold 0.4 + +# Check stats +dev stats + +# Explore patterns +dev explore pattern "error handling" + +# Find similar code +dev explore similar src/utils/retry.ts +``` + +--- + +## Integration Patterns + +### With Cursor + +Cursor automatically detects workspace changes. Just: +1. `dev mcp install --cursor` +2. Restart Cursor +3. Use tools in chat + +### With Claude Code + +```bash +dev mcp install +# Tools available immediately +``` + +### In Scripts + +```bash +# JSON output for scripting +dev search "coordinator" --json | jq '.[].metadata.path' + +# Check if indexed +dev stats --json | jq '.filesIndexed' +``` + +--- + +## Troubleshooting + +### "No results found" + +```bash +# Check if indexed +dev stats + +# Re-index +dev index . +``` + +### "Repository not indexed" + +```bash +dev index . +dev mcp install --cursor +# Restart Cursor +``` + +### Slow responses + +- Reduce `limit` +- Use `compact` format +- Check `dev_health` + +--- + +**More help:** See [Troubleshooting Guide](../TROUBLESHOOTING.md) diff --git a/packages/core/src/map/index.ts b/packages/core/src/map/index.ts index 409b76b..e20c797 100644 --- a/packages/core/src/map/index.ts +++ b/packages/core/src/map/index.ts @@ -79,10 +79,12 @@ function buildDirectoryTree(docs: SearchResult[], opts: Required): M } const dir = path.dirname(filePath); - if (!byDir.has(dir)) { - byDir.set(dir, []); + const existing = byDir.get(dir); + if (existing) { + existing.push(doc); + } else { + byDir.set(dir, [doc]); } - byDir.get(dir)!.push(doc); } // Build tree structure diff --git a/packages/mcp-server/src/adapters/built-in/plan-adapter.ts b/packages/mcp-server/src/adapters/built-in/plan-adapter.ts index ea436e5..7a21c23 100644 --- a/packages/mcp-server/src/adapters/built-in/plan-adapter.ts +++ b/packages/mcp-server/src/adapters/built-in/plan-adapter.ts @@ -6,7 +6,7 @@ */ import type { RepositoryIndexer } from '@lytics/dev-agent-core'; -import type { ContextAssemblyOptions, ContextPackage } from '@lytics/dev-agent-subagents'; +import type { ContextAssemblyOptions } from '@lytics/dev-agent-subagents'; import { assembleContext, formatContextPackage } from '@lytics/dev-agent-subagents'; import { estimateTokensForText, startTimer } from '../../formatters/utils'; import { ToolAdapter } from '../tool-adapter'; diff --git a/packages/subagents/src/planner/utils/__tests__/context-assembler.test.ts b/packages/subagents/src/planner/utils/__tests__/context-assembler.test.ts new file mode 100644 index 0000000..8007cc6 --- /dev/null +++ b/packages/subagents/src/planner/utils/__tests__/context-assembler.test.ts @@ -0,0 +1,368 @@ +import type { RepositoryIndexer, SearchResult } from '@lytics/dev-agent-core'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; +import type { ContextPackage } from '../../context-types'; +import { assembleContext, formatContextPackage } from '../context-assembler'; + +// Mock the GitHub fetch +vi.mock('../github', () => ({ + fetchGitHubIssue: vi.fn(), +})); + +import { fetchGitHubIssue } from '../github'; + +const mockFetchGitHubIssue = vi.mocked(fetchGitHubIssue); + +describe('Context Assembler', () => { + const mockIssue = { + number: 42, + title: 'Add user authentication', + body: 'We need to add JWT-based authentication to the API.\n\n## Acceptance Criteria\n- Login endpoint\n- Logout endpoint', + state: 'open' as const, + createdAt: '2025-01-01T00:00:00Z', + updatedAt: '2025-01-02T00:00:00Z', + labels: ['feature', 'security'], + assignees: [], + author: 'testuser', + comments: [ + { + author: 'reviewer', + body: 'Consider using refresh tokens too', + createdAt: '2025-01-01T12:00:00Z', + }, + ], + }; + + const mockSearchResults: SearchResult[] = [ + { + id: '1', + score: 0.85, + metadata: { + path: 'src/auth/jwt.ts', + name: 'verifyToken', + type: 'function', + snippet: 'export function verifyToken(token: string): boolean { ... }', + file: 'src/auth/jwt.ts', + startLine: 10, + endLine: 20, + exported: true, + }, + }, + { + id: '2', + score: 0.72, + metadata: { + path: 'src/auth/types.ts', + name: 'AuthConfig', + type: 'interface', + snippet: 'export interface AuthConfig { secret: string; }', + file: 'src/auth/types.ts', + startLine: 1, + endLine: 5, + exported: true, + }, + }, + ]; + + const mockIndexer = { + search: vi.fn().mockResolvedValue(mockSearchResults), + } as unknown as RepositoryIndexer; + + beforeEach(() => { + vi.clearAllMocks(); + mockFetchGitHubIssue.mockResolvedValue(mockIssue); + }); + + describe('assembleContext', () => { + it('should assemble a complete context package', async () => { + const result = await assembleContext(42, mockIndexer, '/repo'); + + expect(result.issue.number).toBe(42); + expect(result.issue.title).toBe('Add user authentication'); + expect(result.issue.author).toBe('testuser'); + expect(result.issue.labels).toEqual(['feature', 'security']); + expect(result.issue.comments).toHaveLength(1); + }); + + it('should include relevant code from search', async () => { + const result = await assembleContext(42, mockIndexer, '/repo'); + + expect(result.relevantCode).toHaveLength(2); + expect(result.relevantCode[0].file).toBe('src/auth/jwt.ts'); + expect(result.relevantCode[0].name).toBe('verifyToken'); + expect(result.relevantCode[0].relevanceScore).toBe(0.85); + }); + + it('should skip code search when includeCode is false', async () => { + const result = await assembleContext(42, mockIndexer, '/repo', { + includeCode: false, + includePatterns: false, // Also disable patterns to avoid any search calls + }); + + expect(result.relevantCode).toHaveLength(0); + expect(mockIndexer.search).not.toHaveBeenCalled(); + }); + + it('should handle null indexer gracefully', async () => { + const result = await assembleContext(42, null, '/repo'); + + expect(result.relevantCode).toHaveLength(0); + expect(result.metadata.codeSearchUsed).toBe(false); + }); + + it('should respect maxCodeResults option', async () => { + await assembleContext(42, mockIndexer, '/repo', { + maxCodeResults: 5, + }); + + expect(mockIndexer.search).toHaveBeenCalledWith( + expect.any(String), + expect.objectContaining({ limit: 5 }) + ); + }); + + it('should detect codebase patterns', async () => { + // Mock search to return test files + const testIndexer = { + search: vi.fn().mockResolvedValue([ + { + id: '1', + score: 0.8, + metadata: { + path: 'src/__tests__/auth.test.ts', + name: 'auth tests', + type: 'file', + }, + }, + ]), + } as unknown as RepositoryIndexer; + + const result = await assembleContext(42, testIndexer, '/repo'); + + expect(result.codebasePatterns.testPattern).toBe('*.test.ts'); + expect(result.codebasePatterns.testLocation).toBe('__tests__/'); + }); + + it('should skip pattern detection when includePatterns is false', async () => { + const result = await assembleContext(42, mockIndexer, '/repo', { + includePatterns: false, + }); + + expect(result.codebasePatterns).toEqual({}); + }); + + it('should include metadata with token estimate', async () => { + const result = await assembleContext(42, mockIndexer, '/repo'); + + expect(result.metadata.generatedAt).toBeDefined(); + expect(result.metadata.tokensUsed).toBeGreaterThan(0); + expect(result.metadata.codeSearchUsed).toBe(true); + expect(result.metadata.repositoryPath).toBe('/repo'); + }); + + it('should handle search errors gracefully', async () => { + const errorIndexer = { + search: vi.fn().mockRejectedValue(new Error('Search failed')), + } as unknown as RepositoryIndexer; + + const result = await assembleContext(42, errorIndexer, '/repo'); + + // Should not throw, just return empty code + expect(result.relevantCode).toHaveLength(0); + }); + + it('should infer relevance reasons correctly', async () => { + // Mock issue with title matching a function name + mockFetchGitHubIssue.mockResolvedValueOnce({ + ...mockIssue, + title: 'Fix verifyToken function', + }); + + const result = await assembleContext(42, mockIndexer, '/repo'); + + expect(result.relevantCode[0].reason).toBe('Name matches issue title'); + }); + }); + + describe('formatContextPackage', () => { + const mockContext: ContextPackage = { + issue: { + number: 42, + title: 'Add user authentication', + body: 'We need JWT auth', + labels: ['feature'], + author: 'testuser', + createdAt: '2025-01-01T00:00:00Z', + updatedAt: '2025-01-02T00:00:00Z', + state: 'open', + comments: [ + { + author: 'reviewer', + body: 'Looks good', + createdAt: '2025-01-01T12:00:00Z', + }, + ], + }, + relevantCode: [ + { + file: 'src/auth.ts', + name: 'authenticate', + type: 'function', + snippet: 'function authenticate() {}', + relevanceScore: 0.85, + reason: 'Similar function pattern', + }, + ], + codebasePatterns: { + testPattern: '*.test.ts', + testLocation: '__tests__/', + }, + relatedHistory: [ + { + type: 'pr', + number: 10, + title: 'Previous auth work', + state: 'merged', + relevanceScore: 0.7, + }, + ], + metadata: { + generatedAt: '2025-01-03T00:00:00Z', + tokensUsed: 500, + codeSearchUsed: true, + historySearchUsed: true, + repositoryPath: '/repo', + }, + }; + + it('should format issue header correctly', () => { + const output = formatContextPackage(mockContext); + + expect(output).toContain('# Issue #42: Add user authentication'); + expect(output).toContain('**Author:** testuser'); + expect(output).toContain('**State:** open'); + expect(output).toContain('**Labels:** feature'); + }); + + it('should format issue description', () => { + const output = formatContextPackage(mockContext); + + expect(output).toContain('## Description'); + expect(output).toContain('We need JWT auth'); + }); + + it('should format comments section', () => { + const output = formatContextPackage(mockContext); + + expect(output).toContain('## Comments'); + expect(output).toContain('**reviewer**'); + expect(output).toContain('Looks good'); + }); + + it('should format relevant code section', () => { + const output = formatContextPackage(mockContext); + + expect(output).toContain('## Relevant Code'); + expect(output).toContain('### authenticate (function)'); + expect(output).toContain('**File:** `src/auth.ts`'); + expect(output).toContain('**Relevance:** 85%'); + expect(output).toContain('```typescript'); + expect(output).toContain('function authenticate() {}'); + }); + + it('should format codebase patterns section', () => { + const output = formatContextPackage(mockContext); + + expect(output).toContain('## Codebase Patterns'); + expect(output).toContain('**Test naming:** *.test.ts'); + expect(output).toContain('**Test location:** __tests__/'); + }); + + it('should format related history section', () => { + const output = formatContextPackage(mockContext); + + expect(output).toContain('## Related History'); + expect(output).toContain('**PR #10:** Previous auth work (merged)'); + }); + + it('should include metadata footer', () => { + const output = formatContextPackage(mockContext); + + expect(output).toContain('*Context assembled at'); + expect(output).toContain('~500 tokens*'); + }); + + it('should handle empty comments gracefully', () => { + const contextNoComments: ContextPackage = { + ...mockContext, + issue: { ...mockContext.issue, comments: [] }, + }; + + const output = formatContextPackage(contextNoComments); + + expect(output).not.toContain('## Comments'); + }); + + it('should handle empty code results gracefully', () => { + const contextNoCode: ContextPackage = { + ...mockContext, + relevantCode: [], + }; + + const output = formatContextPackage(contextNoCode); + + expect(output).not.toContain('## Relevant Code'); + }); + + it('should handle empty patterns gracefully', () => { + const contextNoPatterns: ContextPackage = { + ...mockContext, + codebasePatterns: {}, + }; + + const output = formatContextPackage(contextNoPatterns); + + expect(output).not.toContain('## Codebase Patterns'); + }); + + it('should handle empty history gracefully', () => { + const contextNoHistory: ContextPackage = { + ...mockContext, + relatedHistory: [], + }; + + const output = formatContextPackage(contextNoHistory); + + expect(output).not.toContain('## Related History'); + }); + + it('should handle missing description', () => { + const contextNoBody: ContextPackage = { + ...mockContext, + issue: { ...mockContext.issue, body: '' }, + }; + + const output = formatContextPackage(contextNoBody); + + expect(output).toContain('_No description provided_'); + }); + + it('should handle issues type in history', () => { + const contextWithIssue: ContextPackage = { + ...mockContext, + relatedHistory: [ + { + type: 'issue', + number: 5, + title: 'Related bug', + state: 'closed', + relevanceScore: 0.6, + }, + ], + }; + + const output = formatContextPackage(contextWithIssue); + + expect(output).toContain('**Issue #5:** Related bug (closed)'); + }); + }); +}); diff --git a/website/content/docs/_meta.js b/website/content/docs/_meta.js index 2264112..57d91e0 100644 --- a/website/content/docs/_meta.js +++ b/website/content/docs/_meta.js @@ -2,6 +2,9 @@ export default { index: 'Introduction', install: 'Installation', quickstart: 'Quickstart', + cli: 'CLI Reference', + configuration: 'Configuration', + troubleshooting: 'Troubleshooting', '---tools': { type: 'separator', title: 'MCP Tools', diff --git a/website/content/docs/cli.mdx b/website/content/docs/cli.mdx new file mode 100644 index 0000000..c98aba2 --- /dev/null +++ b/website/content/docs/cli.mdx @@ -0,0 +1,290 @@ +# CLI Reference + +The `dev` command-line interface provides local access to dev-agent's code intelligence features. + +## Installation + +```bash +npm install -g dev-agent +``` + +## Commands + +### `dev init` + +Initialize dev-agent in your repository. + +```bash +dev init +``` + +Creates a `.dev-agent.json` configuration file with default settings. + +### `dev index` + +Index your repository for semantic search. + +```bash +dev index . +dev index /path/to/repo +``` + +**Options:** +| Flag | Description | +|------|-------------| +| `-f, --force` | Force re-index even if unchanged | +| `-v, --verbose` | Show verbose output | + +### `dev search` + +Search your indexed code semantically. + +```bash +dev search "authentication logic" +dev search "error handling" --limit 5 --threshold 0.4 +``` + +**Options:** +| Flag | Description | Default | +|------|-------------|---------| +| `-l, --limit ` | Maximum results | 10 | +| `-t, --threshold ` | Minimum similarity (0-1) | 0.7 | +| `--json` | Output as JSON | false | + +**Threshold Guide:** +- `0.7+` — Precise matches only +- `0.4-0.6` — Balanced (good for most searches) +- `0.25-0.3` — Exploratory (finds related concepts) + +### `dev explore` + +Explore code patterns and relationships. + +```bash +# Find patterns semantically +dev explore pattern "error handling" --limit 5 + +# Find code similar to a file +dev explore similar path/to/file.ts +``` + +**Options:** +| Flag | Description | Default | +|------|-------------|---------| +| `-l, --limit ` | Maximum results | 10 | +| `-t, --threshold ` | Minimum similarity | 0.7 | + +### `dev plan` + +Assemble context for implementing a GitHub issue. + +```bash +dev plan 42 +dev plan 42 --include-comments --include-history +``` + +**Options:** +| Flag | Description | Default | +|------|-------------|---------| +| `--include-comments` | Include issue comments | true | +| `--include-history` | Include related PRs/issues | true | +| `--token-budget ` | Max tokens for output | 4000 | + +### `dev gh` + +Index and search GitHub issues/PRs. + +```bash +# Index GitHub metadata +dev gh index + +# Search issues/PRs +dev gh search "authentication bug" +``` + +### `dev update` + +Incrementally update the index with changed files. + +```bash +dev update +``` + +**Options:** +| Flag | Description | +|------|-------------| +| `-v, --verbose` | Show verbose output | + +### `dev stats` + +Show indexing statistics. + +```bash +dev stats +dev stats --json +``` + +**Options:** +| Flag | Description | +|------|-------------| +| `--json` | Output as JSON | + +### `dev clean` + +Remove all indexed data. + +```bash +dev clean --force +``` + +**Options:** +| Flag | Description | +|------|-------------| +| `-f, --force` | Skip confirmation prompt | + +### `dev storage` + +Manage storage locations. + +```bash +dev storage path # Show storage path +dev storage size # Show storage size +``` + +### `dev compact` + +Compact the vector database to reclaim space. + +```bash +dev compact +``` + +--- + +## MCP Commands + +Commands for Model Context Protocol integration with AI tools. + +### `dev mcp start` + +Start the MCP server for AI tool integration. + +```bash +dev mcp start +dev mcp start --verbose +``` + +**Options:** +| Flag | Description | Default | +|------|-------------|---------| +| `-t, --transport ` | Transport type (stdio/http) | stdio | +| `-p, --port ` | Port for HTTP transport | 3000 | +| `-v, --verbose` | Verbose logging | false | + +### `dev mcp install` + +Install dev-agent as an MCP server in Claude Code or Cursor. + +```bash +# Install for Claude Code +dev mcp install + +# Install for Cursor +dev mcp install --cursor +``` + +**Options:** +| Flag | Description | +|------|-------------| +| `-r, --repository ` | Repository path (default: cwd) | +| `--cursor` | Install for Cursor instead of Claude Code | + +### `dev mcp uninstall` + +Remove dev-agent MCP server from Claude Code or Cursor. + +```bash +dev mcp uninstall +dev mcp uninstall --cursor +``` + +### `dev mcp list` + +List all configured MCP servers. + +```bash +dev mcp list +dev mcp list --cursor +``` + +--- + +## Configuration + +The `.dev-agent.json` file configures the indexer: + +```json +{ + "repositoryPath": "/path/to/repo", + "vectorStorePath": ".dev-agent/vectors.lance", + "embeddingModel": "Xenova/all-MiniLM-L6-v2", + "dimension": 384, + "excludePatterns": [ + "**/node_modules/**", + "**/dist/**", + "**/.git/**" + ], + "languages": ["typescript", "javascript", "markdown"] +} +``` + +--- + +## Examples + +### Basic Workflow + +```bash +# Initialize and index +dev init +dev index . + +# Search for code +dev search "user authentication" + +# View statistics +dev stats +``` + +### Integration with Cursor + +```bash +# Index your project +dev index . + +# Install MCP server +dev mcp install --cursor + +# Restart Cursor - tools are now available! +``` + +### Scripting with JSON Output + +```bash +# Get file paths from search +dev search "coordinator" --json | jq '.[].metadata.path' | sort -u + +# Check if indexed +dev stats --json | jq '.filesIndexed' +``` + +### Exploratory Search + +```bash +# Lower threshold for broader matches +dev search "architectural patterns" --threshold 0.25 --limit 10 + +# Find similar code +dev explore similar src/utils/retry.ts --threshold 0.3 +``` + diff --git a/website/content/docs/configuration.mdx b/website/content/docs/configuration.mdx new file mode 100644 index 0000000..0a875d7 --- /dev/null +++ b/website/content/docs/configuration.mdx @@ -0,0 +1,229 @@ +# Configuration + +Dev-agent uses a layered configuration system with sensible defaults. + +## Configuration File + +Run `dev init` to create a configuration file at `.dev-agent/config.json`: + +```json +{ + "version": "1.0", + "repository": { + "path": ".", + "excludePatterns": [ + "**/node_modules/**", + "**/dist/**", + "**/.git/**", + "**/coverage/**" + ], + "languages": ["typescript", "javascript", "markdown"] + }, + "mcp": { + "adapters": { + "search": { "enabled": true }, + "github": { "enabled": true }, + "plan": { "enabled": true }, + "explore": { "enabled": true }, + "status": { "enabled": false } + } + } +} +``` + +## Configuration Options + +### Repository Settings + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `repository.path` | string | `"."` | Path to repository root | +| `repository.excludePatterns` | string[] | See below | Glob patterns to exclude | +| `repository.languages` | string[] | `["typescript", "javascript", "markdown"]` | Languages to index | + +**Default Exclude Patterns:** +- `**/node_modules/**` +- `**/dist/**` +- `**/.git/**` +- `**/coverage/**` + +### MCP Adapter Settings + +Control which MCP tools are enabled: + +| Adapter | Default | Tool | Description | +|---------|---------|------|-------------| +| `search` | enabled | `dev_search` | Semantic code search | +| `github` | enabled | `dev_gh` | GitHub issues/PRs search | +| `plan` | enabled | `dev_plan` | Context assembly for issues | +| `explore` | enabled | `dev_explore` | Code exploration | +| `status` | disabled | `dev_status` | Repository status | +| `refs` | enabled | `dev_refs` | Relationship queries | +| `map` | enabled | `dev_map` | Codebase overview | + +**Disable an adapter:** +```json +{ + "mcp": { + "adapters": { + "github": { "enabled": false } + } + } +} +``` + +## Environment Variables + +Configuration values can reference environment variables using `${VAR_NAME}` syntax: + +```json +{ + "repository": { + "path": "${REPO_PATH}" + } +} +``` + +**Built-in Environment Variables:** + +| Variable | Description | Used By | +|----------|-------------|---------| +| `WORKSPACE_FOLDER_PATHS` | Cursor workspace paths | MCP server | +| `REPOSITORY_PATH` | Explicit repository path | MCP server | + +## Storage Locations + +Dev-agent stores index data in `~/.dev-agent/indexes/`: + +``` +~/.dev-agent/ +└── indexes/ + └── {hash}/ # Per-repository storage + ├── vectors.lance # Vector embeddings + ├── metadata.json # Repository metadata + ├── indexer-state.json + └── github-state.json +``` + +**Storage path resolution:** +1. If repository has a git remote → hash of `owner/repo` +2. Otherwise → hash of absolute path + +This means the same repository cloned to different locations shares the same index. + +### Check Storage Location + +```bash +dev storage path +# Output: ~/.dev-agent/indexes/a1b2c3d4/ + +dev storage size +# Output: 12.5 MB +``` + +## Cursor Integration + +Cursor uses a global MCP config at `~/.cursor/mcp.json`: + +```json +{ + "mcpServers": { + "dev-agent": { + "command": "dev", + "args": ["mcp", "start"] + } + } +} +``` + +**Automatic workspace detection:** +- Cursor sets `WORKSPACE_FOLDER_PATHS` automatically +- Single config works across all projects +- No per-project configuration needed + +Install with: +```bash +dev mcp install --cursor +``` + +## Claude Code Integration + +Claude Code uses the `claude` CLI for MCP configuration: + +```bash +# Install +dev mcp install + +# List servers +claude mcp list + +# Remove +dev mcp uninstall +``` + +## Embedding Model + +Dev-agent uses `Xenova/all-MiniLM-L6-v2` for embeddings: + +| Property | Value | +|----------|-------| +| Model | all-MiniLM-L6-v2 | +| Dimensions | 384 | +| Provider | @xenova/transformers | +| Location | Local (no API calls) | + +The model is downloaded on first run (~23MB) and cached locally. + +## Advanced: Custom Exclude Patterns + +Add project-specific patterns: + +```json +{ + "repository": { + "excludePatterns": [ + "**/node_modules/**", + "**/dist/**", + "**/.git/**", + "**/coverage/**", + "**/vendor/**", + "**/*.generated.ts", + "**/legacy/**" + ] + } +} +``` + +## Troubleshooting + +### Config Not Found + +```bash +# Check if config exists +ls -la .dev-agent/config.json + +# Create config +dev init +``` + +### Environment Variable Not Set + +If you see `Environment variable X is not set`: + +```bash +# Set the variable +export REPO_PATH=/path/to/repo + +# Or use a literal value in config instead +``` + +### Storage Permission Issues + +```bash +# Check permissions +ls -la ~/.dev-agent/ + +# Fix permissions +chmod -R 755 ~/.dev-agent/ +``` + diff --git a/website/content/docs/troubleshooting.mdx b/website/content/docs/troubleshooting.mdx new file mode 100644 index 0000000..11766a2 --- /dev/null +++ b/website/content/docs/troubleshooting.mdx @@ -0,0 +1,161 @@ +# Troubleshooting + +Quick solutions for common issues. For detailed troubleshooting, see the [full guide on GitHub](https://github.com/lytics/dev-agent/blob/main/TROUBLESHOOTING.md). + +## Installation Issues + +### `dev: command not found` + +Global npm bin directory not in PATH: + +```bash +# Find npm bin directory +npm config get prefix + +# Add to shell profile (~/.zshrc or ~/.bashrc) +export PATH="$(npm config get prefix)/bin:$PATH" + +# Reload +source ~/.zshrc +``` + +### Permission errors on install + +```bash +# Option 1: Use npx (no install needed) +npx dev-agent index . + +# Option 2: Fix npm permissions +npm config set prefix ~/.npm-global +export PATH=~/.npm-global/bin:$PATH +npm install -g dev-agent +``` + +## Indexing Issues + +### "No source files found" + +```bash +# Verify you're in repository root +ls -la # Should see .git/ + +# Check for supported files +find . -name "*.ts" -o -name "*.js" | head -10 +``` + +### "Vector storage initialization failed" + +```bash +# Check permissions +ls -la ~/.dev-agent/ + +# Clear and rebuild +rm -rf ~/.dev-agent/indexes/* +dev index . +``` + +## MCP Server Issues + +### Server won't start + +```bash +# 1. Check if indexed +ls -la ~/.dev-agent/indexes/ + +# 2. Verify installation +dev mcp list --cursor # or without --cursor + +# 3. Test manually +dev mcp start --verbose +``` + +### "Repository not indexed" in Cursor + +```bash +# Index the workspace +dev index . + +# Restart Cursor +``` + +### Rate limit errors (429) + +- Wait for `retryAfterMs` period +- Check health: `dev_health` +- Restart AI tool if persistent + +## Search Issues + +### No results + +```bash +# Verify indexed +dev stats + +# Re-index if needed +dev index . +``` + +**Tips for better searches:** +- Use natural language, not exact code +- Describe what code does +- Try different phrasings + +### Poor relevance + +Adjust score threshold: +- `0.7+` — Precise matches only +- `0.4-0.6` — Balanced +- `0.25-0.3` — Exploratory + +## GitHub Integration + +### `dev gh index` fails + +```bash +# Check GitHub CLI +gh auth status + +# Login if needed +gh auth login +``` + +### Stale GitHub data + +```bash +dev gh index # Re-index +``` + +## Quick Fixes + +### Clear everything and start fresh + +```bash +rm -rf ~/.dev-agent/indexes/* +dev index . +dev gh index +dev mcp install --cursor +``` + +### Check health + +``` +Use dev_health tool in Cursor/Claude Code +``` + +### Enable debug logging + +**Cursor:** Edit `~/.cursor/mcp.json`: +```json +"env": { "LOG_LEVEL": "debug" } +``` + +## Getting Help + +1. Run `dev_health` for diagnostics +2. Check [full troubleshooting guide](https://github.com/lytics/dev-agent/blob/main/TROUBLESHOOTING.md) +3. [File an issue](https://github.com/lytics/dev-agent/issues) with: + - `dev --version` + - `dev_health` output + - Steps to reproduce +