Add complete documentation for the new experimental Codebase Indexing feature that enables semantic code search using AI embeddings.

hannesrudolph · hannesrudolph · commit a10a0bbc7be9 · 2025-05-24T12:53:28.000-06:00
## New Documentation
- **docs/features/experimental/codebase-indexing.mdx**: Comprehensive feature documentation covering:
  - Semantic search capabilities using Tree-sitter parsing and AI embeddings
  - Setup requirements for OpenAI/Ollama embedding providers and Qdrant vector database
  - Configuration steps and status indicators
  - File processing with smart code parsing and automatic filtering
  - Best practices for model selection and security considerations
  - Current limitations and future enhancements
  - Privacy and security considerations

## Updated Navigation &amp; Cross-references
- **sidebars.ts**: Added Codebase Indexing to Features &gt; Experimental navigation menu
- **docs/features/experimental/experimental-features.md**:
  - Added Codebase Indexing to experimental features list
  - Added screenshot showing the experimental features settings panel
- **docs/faq.md**: Added FAQ entries explaining:
  - What Codebase Indexing is and its semantic search capabilities
  - Cost considerations for embedding generation and vector storage

## Assets
- **static/img/experimental-features/experimental-features.png**: Screenshot of experimental features settings panel

## Technical Details Covered
- Tree-sitter integration for AST-based code parsing
- Support for both OpenAI and Ollama embedding providers
- Qdrant vector database integration with local and cloud deployment options
- Incremental indexing with file watching and hash-based caching
- Smart file filtering excluding binaries, large files, and common ignore patterns
- codebase_search tool integration for AI-powered code discovery

The documentation is targeted at a semi-technical audience and provides practical setup guidance while explaining the underlying semantic search technology.
diff --git a/docs/faq.md b/docs/faq.md
@@ -136,6 +136,14 @@ Yes, if you use a [local model](/advanced-usage/local-models).
 
 Yes, you can create your own MCP servers to add custom functionality to Roo Code. See the [MCP documentation](https://github.com/modelcontextprotocol) for details.
 
+### What is Codebase Indexing?
+
+[Codebase Indexing](/features/experimental/codebase-indexing) is an experimental feature that creates a semantic search index of your project using AI embeddings. This enables Roo Code to better understand and navigate large codebases by finding relevant code based on meaning rather than just keywords.
+
+### How much does Codebase Indexing cost?
+
+Codebase Indexing requires an OpenAI API key for generating embeddings and a Qdrant vector database for storage. Costs depend on your project size and the embedding model used. Initial indexing is the most expensive part; subsequent updates are incremental and much cheaper.
+
 ## Troubleshooting
 
 ### Roo Code isn't responding. What should I do?
@@ -148,7 +156,7 @@ Yes, you can create your own MCP servers to add custom functionality to Roo Code
 
 ### I'm seeing an error message. What does it mean?
 
-The error message should provide some information about the problem. If you're unsure how to resolve it, seek help in the community forums.
+The error message should provide some information about the problem. If you're unsure how to resolve it, seek help in [Discord](https://discord.gg/roocode).
 
 ### Roo Code made changes I didn't want. How do I undo them?
 
diff --git a/docs/features/experimental/codebase-indexing.mdx b/docs/features/experimental/codebase-indexing.mdx
@@ -0,0 +1,188 @@
+import Codicon from '@site/src/components/Codicon';
+
+# Codebase Indexing
+
+**⚠️ Experimental Feature:** This feature is under active development and may change significantly in future releases.
+
+Codebase Indexing enables semantic code search across your entire project using AI embeddings. Instead of searching for exact text matches, it understands the *meaning* of your queries, helping Roo Code find relevant code even when you don't know specific function names or file locations.
+
+<img src="/img/experimental-features/experimental-features.png" alt="Codebase Indexing Settings" width="800" />
+
+## What It Does
+
+When enabled, the indexing system:
+
+1. **Parses your code** using Tree-sitter to identify semantic blocks (functions, classes, methods)
+2. **Creates embeddings** of each code block using AI models
+3. **Stores vectors** in a Qdrant database for fast similarity search
+4. **Provides the `codebase_search` tool** to Roo for intelligent code discovery
+
+This enables natural language queries like "user authentication logic" or "database connection handling" to find relevant code across your entire project.
+
+## Key Benefits
+
+- **Semantic Search**: Find code by meaning, not just keywords
+- **Enhanced AI Understanding**: Roo can better comprehend and work with your codebase
+- **Cross-Project Discovery**: Search across all files, not just what's open
+- **Pattern Recognition**: Locate similar implementations and code patterns
+
+## Setup Requirements
+
+### Embedding Provider
+
+Choose one of these options for generating embeddings:
+
+**OpenAI (Recommended)**
+- Requires OpenAI API key
+- Supports all OpenAI embedding models
+- Default: `text-embedding-3-small`
+- Processes up to 100,000 tokens per batch
+
+**Ollama (Local)**
+- Requires local Ollama installation
+- No API costs or internet dependency
+- Supports any Ollama-compatible embedding model
+- Requires Ollama base URL configuration
+
+### Vector Database
+
+**Qdrant** is required for storing and searching embeddings:
+- **Local**: `http://localhost:6333` (recommended for testing)
+- **Cloud**: Qdrant Cloud or self-hosted instance
+- **Authentication**: Optional API key for secured deployments
+
+## Setting Up Qdrant
+
+### Quick Local Setup
+
+**Using Docker:**
+```bash
+docker run -p 6333:6333 qdrant/qdrant
+```
+
+**Using Docker Compose:**
+```yaml
+version: '3.8'
+services:
+  qdrant:
+    image: qdrant/qdrant
+    ports:
+      - "6333:6333"
+    volumes:
+      - qdrant_storage:/qdrant/storage
+volumes:
+  qdrant_storage:
+```
+
+### Production Deployment
+
+For team or production use:
+- [Qdrant Cloud](https://cloud.qdrant.io/) - Managed service
+- Self-hosted on AWS, GCP, or Azure
+- Local server with network access for team sharing
+
+## Configuration
+
+1. Open Roo Code settings (<Codicon name="gear" /> icon)
+2. Navigate to **Experimental** section
+3. Enable **"Enable Codebase Indexing"**
+4. Configure your embedding provider:
+   - **OpenAI**: Enter API key and select model
+   - **Ollama**: Enter base URL and select model
+5. Set Qdrant URL and optional API key
+6. Click **Save** to start initial indexing
+
+## Understanding Index Status
+
+The interface shows real-time status with color indicators:
+
+- **Standby** (Gray): Not running, awaiting configuration
+- **Indexing** (Yellow): Currently processing files
+- **Indexed** (Green): Up-to-date and ready for searches
+- **Error** (Red): Failed state requiring attention
+
+## How Files Are Processed
+
+### Smart Code Parsing
+- **Tree-sitter Integration**: Uses AST parsing to identify semantic code blocks
+- **Language Support**: All languages supported by Tree-sitter
+- **Fallback**: Line-based chunking for unsupported file types
+- **Block Sizing**: 
+  - Minimum: 100 characters
+  - Maximum: 1,000 characters
+  - Splits large functions intelligently
+
+### Automatic File Filtering
+The indexer automatically excludes:
+- Binary files and images
+- Large files (&gt;1MB)
+- Git repositories (`.git` folders)
+- Dependencies (`node_modules`, `vendor`, etc.)
+- Files matching `.gitignore` and `.rooignore` patterns
+
+### Incremental Updates
+- **File Watching**: Monitors workspace for changes
+- **Smart Updates**: Only reprocesses modified files
+- **Hash-based Caching**: Avoids reprocessing unchanged content
+- **Branch Switching**: Automatically handles Git branch changes
+
+## Best Practices
+
+### Model Selection
+
+**For OpenAI:**
+- **`text-embedding-3-small`**: Best balance of performance and cost
+- **`text-embedding-3-large`**: Higher accuracy, 5x more expensive
+- **`text-embedding-ada-002`**: Legacy model, lower cost
+
+**For Ollama:**
+- Choose models based on your hardware capabilities
+- Larger models provide better accuracy but require more resources
+
+### Security Considerations
+- **API Keys**: Stored securely in VS Code's encrypted storage
+- **Code Privacy**: Only small code snippets sent for embedding (not full files)
+- **Local Processing**: All parsing happens locally
+- **Qdrant Security**: Use authentication for production deployments
+
+## Current Limitations
+
+- **File Size**: 1MB maximum per file
+- **Markdown**: Not currently supported due to parsing complexity
+- **Single Workspace**: One workspace at a time
+- **Dependencies**: Requires external services (embedding provider + Qdrant)
+- **Language Coverage**: Limited to Tree-sitter supported languages
+
+## Using the Search Feature
+
+Once indexed, Roo can use the `codebase_search` tool to find relevant code:
+
+**Example Queries:**
+- "How is user authentication handled?"
+- "Database connection setup"
+- "Error handling patterns"
+- "API endpoint definitions"
+
+The tool provides Roo with:
+- Relevant code snippets
+- File paths and line numbers
+- Similarity scores
+- Contextual information
+
+## Privacy & Security
+
+- **Code stays local**: Only small code snippets sent for embedding
+- **Embeddings are numeric**: Not human-readable representations
+- **Secure storage**: API keys encrypted in VS Code storage
+- **Local option**: Use Ollama for completely local processing
+- **Access control**: Respects existing file permissions
+
+## Future Enhancements
+
+Planned improvements:
+- Additional embedding providers
+- Improved markdown and documentation support
+- Multi-workspace indexing
+- Enhanced filtering and configuration options
+- Team sharing capabilities
+- Integration with VS Code's native search
diff --git a/docs/features/experimental/experimental-features.md b/docs/features/experimental/experimental-features.md
@@ -5,7 +5,7 @@ Roo Code includes experimental features that are still under development. These
 **Warning:** Experimental features may have unexpected behavior, including potential data loss or security vulnerabilities. Enable them at your own risk.
 
 ## Enabling Experimental Features
-
+![alt text](../../../static/img/experimental-features/experimental-features.png)
 To enable or disable experimental features:
 
 1.  Open the Roo Code settings (<Codicon name="gear" /> icon in the top right corner).
@@ -16,6 +16,7 @@ To enable or disable experimental features:
 
 The following experimental features are currently available:
 
+- [Codebase Indexing](/features/experimental/codebase-indexing) - Semantic search through AI-powered codebase indexing
 - [Intelligently Condense the Context Window](/features/experimental/intelligent-context-condensing)
 - [Power Steering](/features/experimental/power-steering)
 
diff --git a/sidebars.ts b/sidebars.ts
@@ -60,6 +60,7 @@ const sidebars: SidebarsConfig = {
           label: 'Experimental',
           items: [
             'features/experimental/experimental-features',
+            'features/experimental/codebase-indexing',
             'features/experimental/intelligent-context-condensing',
             'features/experimental/power-steering',
           ],
diff --git a/static/img/experimental-features/experimental-features.png b/static/img/experimental-features/experimental-features.png
