Codebase indexing

[janpaul123]

Codebase indexing is the idea of storing some sort of compressed representation of the codebase which can be used by the agent. For example, Cursor and Windsurf generate an embedding for each file in the codebase.

There are open source implementations which we could use:

• This PR to Roo uses OpenAI embeddings and Qdrant. It appears it might soon get merged though, so we could see if we can help get it over the finish line? It does currently require you to use a Cloud-based Qdrant URL, which is not great—a local-first solution would be better to start (maybe later with optional cloud storage for large codebases, or to share it between multiple users).
• Continue.dev also has an open source implementation.
  • There happens to also be a closed Roo PR based on the Continue.dev implementation.

Related tickets:

• Add features from the Continue project: Autocomplete, Embeding indexing, Rerank cline/cline#2465
• Indexing Code Base RooCodeInc/Roo-Code#411
• Integrated RAG / Indexing / vectors like Cursor Indexing or Continue RooCodeInc/Roo-Code#2819
• Automatic Contexting? RooCodeInc/Roo-Code#2854
• Add RAG integration cline/cline#1255
• Add RAG Support for Enhanced Codebase Understanding cline/cline#2642
• Vector database cline/cline#1333
• Why not add vectorization to reduce token occupancy? Now, it directly uploads all code files cline/cline#1097

Keywords: embedding, vector database, RAG (retrieval-augmented generation)

[ofou]

From what I've seen, both Cursor and Windsurf implemented this as an MCP server, and they use this minimal structure when doing semantic search, they use a query and a dir over which to make the search, and an optional explanation param.

I created a minimal implementation as MCP server, using Ollama with a tiny embedding model check https://github.com/ofou/codebase_search, the former implementation seemed too complicated as the first implementation of this, and I don't like that they use a cloud-based vector store.

Semantic search results (raw JSON)

[
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/rag_systems.txt",
    "similarity": 0.6941935269156939,
    "relative_path": "folder_to_test/rag_systems.txt"
  },
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/rag_implementation_code.txt",
    "similarity": 0.6449619035006291,
    "relative_path": "folder_to_test/rag_implementation_code.txt"
  },
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/vector_databases.txt",
    "similarity": 0.6184170305394026,
    "relative_path": "folder_to_test/vector_databases.txt"
  },
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/embedding_models.txt",
    "similarity": 0.47589052901500284,
    "relative_path": "folder_to_test/embedding_models.txt"
  },
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/ml_healthcare.txt",
    "similarity": 0.4476328745171523,
    "relative_path": "folder_to_test/ml_healthcare.txt"
  },
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/document_chunking.txt",
    "similarity": 0.44762066545544027,
    "relative_path": "folder_to_test/document_chunking.txt"
  },
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/climate_change.txt",
    "similarity": 0.4022195935701077,
    "relative_path": "folder_to_test/climate_change.txt"
  },
  {
    "file": "/Users/ofou/os/codebase_search/folder_to_test/rag_evaluation.txt",
    "similarity": 0.38974273918673374,
    "relative_path": "folder_to_test/rag_evaluation.txt"
  }
]

[Juice10]

RooCodeInc/Roo-Code#411 was merged and added indexing in Kilo Code