docs: comprehensive documentation and changeset for v0.6.1 release (#141)

prosdev · web-flow · commit b675fc9ebc33 · 2025-12-11T09:12:32.000-08:00
- Add missing 'dev update' command to CLI documentation (README and website)
- Document performance tuning environment variables in configuration
- Add timing expectations for indexing (5-10 minutes for large codebases)
- Create v0.6.1 release notes covering reliability and performance improvements
- Update website latest version to v0.6.1
- Add comprehensive changeset for patch release covering:
  * Performance configuration with environment variables
  * Enhanced Go scanner reliability and WASM validation
  * Better error handling and logging across scanners
  * 57 new tests and architectural improvements
  * Documentation enhancements and user experience improvements

This properly documents all the improvements merged in the recent comprehensive
Go support enhancement PR, categorized as a patch release (0.6.1) since it
improves the Go feature introduced in 0.6.0 rather than adding new features.
diff --git a/.changeset/document-dev-update-command.md b/.changeset/document-dev-update-command.md
@@ -0,0 +1,36 @@
+---
+"@lytics/dev-agent-cli": patch
+"@lytics/dev-agent": patch
+"@lytics/dev-agent-core": patch
+---
+
+fix: improve reliability, performance, and documentation for Go support
+
+## Major Features
+- **Performance Configuration**: Environment variables for fine-tuning concurrency (DEV_AGENT_*_CONCURRENCY)
+- **Enhanced Go Scanner**: Runtime WASM validation, improved error handling, better reliability
+- **TypeScript Improvements**: Streamlined error handling, better type checking, enhanced progress reporting
+- **System Resource Detection**: Intelligent performance defaults based on CPU and memory
+- **Architectural Utilities**: Reusable modules for WASM resolution, concurrency, and file validation
+
+## New Environment Variables
+- `DEV_AGENT_TYPESCRIPT_CONCURRENCY`: Control TypeScript scanner parallelism
+- `DEV_AGENT_INDEXER_CONCURRENCY`: Configure embedding batch processing
+- `DEV_AGENT_GO_CONCURRENCY`: Tune Go scanner performance
+- `DEV_AGENT_CONCURRENCY`: General fallback for all scanners
+
+## Documentation & User Experience
+- Document missing `dev update` command for incremental indexing
+- Add timing expectations (5-10 minutes for large codebases)
+- Create LANGUAGE_SUPPORT.md contributor guide
+- Enhanced troubleshooting and configuration sections
+- Remove Renovate automation for manual dependency control
+
+## Technical Improvements
+- 57 new tests with comprehensive coverage
+- Dependency injection for testable file system operations
+- Centralized error handling patterns across scanners
+- Build script reliability fixes (prevent silent failures)
+
+This release significantly improves performance, reliability, and developer experience while maintaining backward compatibility.
+
diff --git a/README.md b/README.md
@@ -50,14 +50,17 @@ We benchmarked dev-agent against baseline Claude Code across 5 task types:
 # Install globally
 npm install -g dev-agent
 
-# Index your repository
+# Index your repository (initial indexing can take 5-10 minutes for large codebases)
 cd /path/to/your/repo
 dev index .
 
 # Install MCP integration
 dev mcp install --cursor  # For Cursor IDE
 dev mcp install           # For Claude Code
 
+# Keep index up to date (fast incremental updates)
+dev update
+
 # That's it! AI tools now have access to dev-agent capabilities.
 ```
 
@@ -186,6 +189,7 @@ Check MCP server and component health.
 ### Production Ready
 - 🛡️ Rate limiting (100 req/min burst)
 - 🔄 Retry logic with exponential backoff
+- ⚡ Incremental indexing (only processes changed files)
 - 💚 Health monitoring
 - 🧹 Memory-safe (circular buffers)
 - ✅ 1300+ tests
@@ -223,10 +227,14 @@ npm link
 ## CLI Commands
 
 ```bash
-# Index everything (code, git history, GitHub)
+# Index everything (code, git history, GitHub) - can take 5-10 min for large codebases
 dev index .
 dev index . --no-github               # Skip GitHub indexing
 
+# Incremental updates (only changed files) - much faster, typically seconds
+dev update                            # Fast incremental reindexing
+dev update -v                         # Verbose output showing what changed
+
 # Semantic search
 dev search "how do agents communicate"
 dev search "error handling" --threshold 0.3
@@ -289,6 +297,7 @@ To add new languages, see [LANGUAGE_SUPPORT.md](LANGUAGE_SUPPORT.md).
 
 **Indexing too slow:**
 ```bash
+# Note: Initial indexing can take 5-10 minutes for mature codebases (4k+ files)
 # Try increasing concurrency (if you have enough memory)
 export DEV_AGENT_CONCURRENCY=20
 dev index .
@@ -303,6 +312,14 @@ export DEV_AGENT_INDEXER_CONCURRENCY=2
 dev index .
 ```
 
+**Search results are outdated:**
+```bash
+# Update index with recent file changes
+dev update
+# Or do a full reindex if needed
+dev index .
+```
+
 **Go scanner not working:**
 ```bash
 # Check if WASM files are bundled (after installation/build)
diff --git a/website/content/docs/cli.mdx b/website/content/docs/cli.mdx
@@ -24,6 +24,8 @@ Creates a `.dev-agent.json` configuration file with default settings.
 
 Index your repository for semantic search. Indexes code, git history, and GitHub issues/PRs.
 
+*Note: Initial indexing can take 5-10 minutes for large codebases (4k+ files).*
+
 ```bash
 dev index .
 dev index /path/to/repo
@@ -153,16 +155,17 @@ dev git stats
 
 ### `dev update`
 
-Incrementally update the index with changed files.
+Incrementally update the index with changed files. Much faster than full indexing - typically completes in seconds.
 
 ```bash
 dev update
+dev update -v  # Show what files changed
 ```
 
 **Options:**
 | Flag | Description |
 |------|-------------|
-| `-v, --verbose` | Show verbose output |
+| `-v, --verbose` | Show verbose output and which files changed |
 
 ### `dev stats`
 
diff --git a/website/content/docs/configuration.mdx b/website/content/docs/configuration.mdx
@@ -91,6 +91,43 @@ Configuration values can reference environment variables using `${VAR_NAME}` syn
 | `WORKSPACE_FOLDER_PATHS` | Cursor workspace paths | MCP server |
 | `REPOSITORY_PATH` | Explicit repository path | MCP server |
 
+## Performance Tuning
+
+Control scanning and indexing performance using environment variables:
+
+| Variable | Description | Default | Recommended Range |
+|----------|-------------|---------|-------------------|
+| `DEV_AGENT_CONCURRENCY` | Global concurrency setting (applies to all operations) | Auto-detected | 5-30 |
+| `DEV_AGENT_TYPESCRIPT_CONCURRENCY` | TypeScript file processing concurrency | Auto-detected | 5-40 |
+| `DEV_AGENT_INDEXER_CONCURRENCY` | Vector embedding batch concurrency | Auto-detected | 2-10 |
+| `DEV_AGENT_GO_CONCURRENCY` | Go file processing concurrency | Auto-detected | 5-30 |
+
+**Auto-detection:** If no environment variables are set, dev-agent automatically detects optimal concurrency based on your system's CPU and memory.
+
+### Performance Recommendations
+
+| System Type | Global | TypeScript | Indexer | Notes |
+|-------------|--------|------------|---------|-------|
+| Low memory (<4GB) | 5 | 5 | 2 | Prevents OOM errors |
+| Standard (4-8GB) | 15 | 15 | 3 | Balanced performance |
+| High-end (8GB+, 8+ cores) | 30 | 30 | 5 | Maximum speed |
+
+### Usage Examples
+
+**High-performance settings:**
+```bash
+export DEV_AGENT_TYPESCRIPT_CONCURRENCY=30
+export DEV_AGENT_INDEXER_CONCURRENCY=8
+dev index .
+```
+
+**Memory-conservative settings:**
+```bash
+export DEV_AGENT_CONCURRENCY=5
+export DEV_AGENT_INDEXER_CONCURRENCY=2
+dev index .
+```
+
 ## Storage Locations
 
 Dev-agent stores index data in `~/.dev-agent/indexes/`:
diff --git a/website/content/latest-version.ts b/website/content/latest-version.ts
@@ -4,10 +4,10 @@
  */
 
 export const latestVersion = {
-  version: '0.6.0',
-  title: 'Go Language Support',
-  date: 'December 10, 2025',
+  version: '0.6.1',
+  title: 'Reliability & Performance Improvements',
+  date: 'December 11, 2025',
   summary:
-    'Full Go language support with tree-sitter — functions, methods, structs, interfaces, and generics.',
-  link: '/updates#v060--go-language-support',
+    'Enhanced reliability, performance configuration, and better documentation for Go support.',
+  link: '/updates#v061--reliability--performance-improvements',
 } as const;
diff --git a/website/content/updates/index.mdx b/website/content/updates/index.mdx
@@ -9,6 +9,66 @@ What's new in dev-agent. We ship improvements regularly to help AI assistants un
 
 ---
 
+## v0.6.1 — Reliability & Performance Improvements
+
+*December 11, 2025*
+
+**Enhanced reliability, performance configuration, and better documentation for the Go support released in v0.6.0.**
+
+### What's New
+
+**⚡ Performance Configuration**
+
+Fine-tune scanning and indexing performance with environment variables:
+
+```bash
+# High-performance systems (8GB+ RAM, 8+ cores)
+export DEV_AGENT_TYPESCRIPT_CONCURRENCY=30
+export DEV_AGENT_INDEXER_CONCURRENCY=8
+
+# Memory-conservative systems (<4GB RAM)
+export DEV_AGENT_CONCURRENCY=5
+export DEV_AGENT_INDEXER_CONCURRENCY=2
+
+dev index .  # Uses your custom settings
+```
+
+**Auto-detection:** If no variables are set, dev-agent automatically detects optimal settings based on your CPU and memory.
+
+**🛠️ Enhanced Reliability**
+
+- **Runtime WASM validation** — Go scanner now validates tree-sitter WASM availability before parsing
+- **Build script improvements** — Prevent silent failures that could break Go support
+- **Better error handling** — Clearer error messages and proper logging levels across all scanners
+
+**📚 Documentation Improvements**
+
+- Document missing `dev update` command for fast incremental indexing
+- Add timing expectations: initial indexing takes 5-10 minutes for large codebases (4k+ files)
+- Enhanced troubleshooting guide with performance tuning recommendations
+- New performance tuning section in configuration docs
+
+**🧪 Testing & Architecture**
+
+- 57 new tests covering utility functions and edge cases
+- Dependency injection for testable file system operations
+- Extracted reusable modules for WASM resolution, concurrency calculation, and file validation
+
+### Migration
+
+No breaking changes. All improvements are backward compatible.
+
+### Environment Variables
+
+| Variable | Description | Auto-Detected Default |
+|----------|-------------|----------------------|
+| `DEV_AGENT_CONCURRENCY` | Global concurrency setting | Based on CPU/memory |
+| `DEV_AGENT_TYPESCRIPT_CONCURRENCY` | TypeScript file processing | Based on system resources |
+| `DEV_AGENT_INDEXER_CONCURRENCY` | Vector embedding batches | Based on available memory |
+| `DEV_AGENT_GO_CONCURRENCY` | Go file processing | Based on system resources |
+
+---
+
 ## v0.6.0 — Go Language Support
 
 *December 10, 2025*
