diff --git a/.changeset/document-dev-update-command.md b/.changeset/document-dev-update-command.md new file mode 100644 index 0000000..f6f2894 --- /dev/null +++ b/.changeset/document-dev-update-command.md @@ -0,0 +1,36 @@ +--- +"@lytics/dev-agent-cli": patch +"@lytics/dev-agent": patch +"@lytics/dev-agent-core": patch +--- + +fix: improve reliability, performance, and documentation for Go support + +## Major Features +- **Performance Configuration**: Environment variables for fine-tuning concurrency (DEV_AGENT_*_CONCURRENCY) +- **Enhanced Go Scanner**: Runtime WASM validation, improved error handling, better reliability +- **TypeScript Improvements**: Streamlined error handling, better type checking, enhanced progress reporting +- **System Resource Detection**: Intelligent performance defaults based on CPU and memory +- **Architectural Utilities**: Reusable modules for WASM resolution, concurrency, and file validation + +## New Environment Variables +- `DEV_AGENT_TYPESCRIPT_CONCURRENCY`: Control TypeScript scanner parallelism +- `DEV_AGENT_INDEXER_CONCURRENCY`: Configure embedding batch processing +- `DEV_AGENT_GO_CONCURRENCY`: Tune Go scanner performance +- `DEV_AGENT_CONCURRENCY`: General fallback for all scanners + +## Documentation & User Experience +- Document missing `dev update` command for incremental indexing +- Add timing expectations (5-10 minutes for large codebases) +- Create LANGUAGE_SUPPORT.md contributor guide +- Enhanced troubleshooting and configuration sections +- Remove Renovate automation for manual dependency control + +## Technical Improvements +- 57 new tests with comprehensive coverage +- Dependency injection for testable file system operations +- Centralized error handling patterns across scanners +- Build script reliability fixes (prevent silent failures) + +This release significantly improves performance, reliability, and developer experience while maintaining backward compatibility. + diff --git a/README.md b/README.md index 9eec8b7..5091aec 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ We benchmarked dev-agent against baseline Claude Code across 5 task types: # Install globally npm install -g dev-agent -# Index your repository +# Index your repository (initial indexing can take 5-10 minutes for large codebases) cd /path/to/your/repo dev index . @@ -58,6 +58,9 @@ dev index . dev mcp install --cursor # For Cursor IDE dev mcp install # For Claude Code +# Keep index up to date (fast incremental updates) +dev update + # That's it! AI tools now have access to dev-agent capabilities. ``` @@ -186,6 +189,7 @@ Check MCP server and component health. ### Production Ready - ๐Ÿ›ก๏ธ Rate limiting (100 req/min burst) - ๐Ÿ”„ Retry logic with exponential backoff +- โšก Incremental indexing (only processes changed files) - ๐Ÿ’š Health monitoring - ๐Ÿงน Memory-safe (circular buffers) - โœ… 1300+ tests @@ -223,10 +227,14 @@ npm link ## CLI Commands ```bash -# Index everything (code, git history, GitHub) +# Index everything (code, git history, GitHub) - can take 5-10 min for large codebases dev index . dev index . --no-github # Skip GitHub indexing +# Incremental updates (only changed files) - much faster, typically seconds +dev update # Fast incremental reindexing +dev update -v # Verbose output showing what changed + # Semantic search dev search "how do agents communicate" dev search "error handling" --threshold 0.3 @@ -289,6 +297,7 @@ To add new languages, see [LANGUAGE_SUPPORT.md](LANGUAGE_SUPPORT.md). **Indexing too slow:** ```bash +# Note: Initial indexing can take 5-10 minutes for mature codebases (4k+ files) # Try increasing concurrency (if you have enough memory) export DEV_AGENT_CONCURRENCY=20 dev index . @@ -303,6 +312,14 @@ export DEV_AGENT_INDEXER_CONCURRENCY=2 dev index . ``` +**Search results are outdated:** +```bash +# Update index with recent file changes +dev update +# Or do a full reindex if needed +dev index . +``` + **Go scanner not working:** ```bash # Check if WASM files are bundled (after installation/build) diff --git a/website/content/docs/cli.mdx b/website/content/docs/cli.mdx index 573e7de..9521ec0 100644 --- a/website/content/docs/cli.mdx +++ b/website/content/docs/cli.mdx @@ -24,6 +24,8 @@ Creates a `.dev-agent.json` configuration file with default settings. Index your repository for semantic search. Indexes code, git history, and GitHub issues/PRs. +*Note: Initial indexing can take 5-10 minutes for large codebases (4k+ files).* + ```bash dev index . dev index /path/to/repo @@ -153,16 +155,17 @@ dev git stats ### `dev update` -Incrementally update the index with changed files. +Incrementally update the index with changed files. Much faster than full indexing - typically completes in seconds. ```bash dev update +dev update -v # Show what files changed ``` **Options:** | Flag | Description | |------|-------------| -| `-v, --verbose` | Show verbose output | +| `-v, --verbose` | Show verbose output and which files changed | ### `dev stats` diff --git a/website/content/docs/configuration.mdx b/website/content/docs/configuration.mdx index 0a875d7..167ed0f 100644 --- a/website/content/docs/configuration.mdx +++ b/website/content/docs/configuration.mdx @@ -91,6 +91,43 @@ Configuration values can reference environment variables using `${VAR_NAME}` syn | `WORKSPACE_FOLDER_PATHS` | Cursor workspace paths | MCP server | | `REPOSITORY_PATH` | Explicit repository path | MCP server | +## Performance Tuning + +Control scanning and indexing performance using environment variables: + +| Variable | Description | Default | Recommended Range | +|----------|-------------|---------|-------------------| +| `DEV_AGENT_CONCURRENCY` | Global concurrency setting (applies to all operations) | Auto-detected | 5-30 | +| `DEV_AGENT_TYPESCRIPT_CONCURRENCY` | TypeScript file processing concurrency | Auto-detected | 5-40 | +| `DEV_AGENT_INDEXER_CONCURRENCY` | Vector embedding batch concurrency | Auto-detected | 2-10 | +| `DEV_AGENT_GO_CONCURRENCY` | Go file processing concurrency | Auto-detected | 5-30 | + +**Auto-detection:** If no environment variables are set, dev-agent automatically detects optimal concurrency based on your system's CPU and memory. + +### Performance Recommendations + +| System Type | Global | TypeScript | Indexer | Notes | +|-------------|--------|------------|---------|-------| +| Low memory (<4GB) | 5 | 5 | 2 | Prevents OOM errors | +| Standard (4-8GB) | 15 | 15 | 3 | Balanced performance | +| High-end (8GB+, 8+ cores) | 30 | 30 | 5 | Maximum speed | + +### Usage Examples + +**High-performance settings:** +```bash +export DEV_AGENT_TYPESCRIPT_CONCURRENCY=30 +export DEV_AGENT_INDEXER_CONCURRENCY=8 +dev index . +``` + +**Memory-conservative settings:** +```bash +export DEV_AGENT_CONCURRENCY=5 +export DEV_AGENT_INDEXER_CONCURRENCY=2 +dev index . +``` + ## Storage Locations Dev-agent stores index data in `~/.dev-agent/indexes/`: diff --git a/website/content/latest-version.ts b/website/content/latest-version.ts index 228af70..b4badde 100644 --- a/website/content/latest-version.ts +++ b/website/content/latest-version.ts @@ -4,10 +4,10 @@ */ export const latestVersion = { - version: '0.6.0', - title: 'Go Language Support', - date: 'December 10, 2025', + version: '0.6.1', + title: 'Reliability & Performance Improvements', + date: 'December 11, 2025', summary: - 'Full Go language support with tree-sitter โ€” functions, methods, structs, interfaces, and generics.', - link: '/updates#v060--go-language-support', + 'Enhanced reliability, performance configuration, and better documentation for Go support.', + link: '/updates#v061--reliability--performance-improvements', } as const; diff --git a/website/content/updates/index.mdx b/website/content/updates/index.mdx index b8cb653..0c9b9e4 100644 --- a/website/content/updates/index.mdx +++ b/website/content/updates/index.mdx @@ -9,6 +9,66 @@ What's new in dev-agent. We ship improvements regularly to help AI assistants un --- +## v0.6.1 โ€” Reliability & Performance Improvements + +*December 11, 2025* + +**Enhanced reliability, performance configuration, and better documentation for the Go support released in v0.6.0.** + +### What's New + +**โšก Performance Configuration** + +Fine-tune scanning and indexing performance with environment variables: + +```bash +# High-performance systems (8GB+ RAM, 8+ cores) +export DEV_AGENT_TYPESCRIPT_CONCURRENCY=30 +export DEV_AGENT_INDEXER_CONCURRENCY=8 + +# Memory-conservative systems (<4GB RAM) +export DEV_AGENT_CONCURRENCY=5 +export DEV_AGENT_INDEXER_CONCURRENCY=2 + +dev index . # Uses your custom settings +``` + +**Auto-detection:** If no variables are set, dev-agent automatically detects optimal settings based on your CPU and memory. + +**๐Ÿ› ๏ธ Enhanced Reliability** + +- **Runtime WASM validation** โ€” Go scanner now validates tree-sitter WASM availability before parsing +- **Build script improvements** โ€” Prevent silent failures that could break Go support +- **Better error handling** โ€” Clearer error messages and proper logging levels across all scanners + +**๐Ÿ“š Documentation Improvements** + +- Document missing `dev update` command for fast incremental indexing +- Add timing expectations: initial indexing takes 5-10 minutes for large codebases (4k+ files) +- Enhanced troubleshooting guide with performance tuning recommendations +- New performance tuning section in configuration docs + +**๐Ÿงช Testing & Architecture** + +- 57 new tests covering utility functions and edge cases +- Dependency injection for testable file system operations +- Extracted reusable modules for WASM resolution, concurrency calculation, and file validation + +### Migration + +No breaking changes. All improvements are backward compatible. + +### Environment Variables + +| Variable | Description | Auto-Detected Default | +|----------|-------------|----------------------| +| `DEV_AGENT_CONCURRENCY` | Global concurrency setting | Based on CPU/memory | +| `DEV_AGENT_TYPESCRIPT_CONCURRENCY` | TypeScript file processing | Based on system resources | +| `DEV_AGENT_INDEXER_CONCURRENCY` | Vector embedding batches | Based on available memory | +| `DEV_AGENT_GO_CONCURRENCY` | Go file processing | Based on system resources | + +--- + ## v0.6.0 โ€” Go Language Support *December 10, 2025*