Add code search reranking support via external services

[AbinandN]

Problem Description

Current codebase indexing and search suffers from critical retrieval quality issues that degrade the AI's ability to provide accurate responses:

• Context Pollution: Vector similarity alone retrieves syntactically similar but semantically irrelevant code snippets, polluting the context window with noise. For example, searching for "authentication logic" might retrieve every file that mentions "auth" but miss the actual authentication implementation.

• Who is affected: All users relying on AI-assisted code understanding and generation

• When it happens: During any codebase search operation, especially in large repositories

• Current behavior: Codebase search retrieves top-k results based solely on vector similarity, leading to:

  • Semantically irrelevant results ranking high due to keyword overlap
  • Critical implementation details being pushed out by superficial matches
  • AI receiving diluted context that reduces response accuracy

• Expected behavior: Retrieved context should be semantically relevant to the query intent, not just vector-similar

• Impact:

  • AI provides less accurate responses due to poor context quality
  • Users must repeatedly refine queries to get relevant results
  • Reduced trust in AI suggestions when based on irrelevant context

Without reranking, our codebase search treats all vector-similar results equally, ignoring semantic relevance. This leads to context windows filled with marginally relevant code that drowns out the truly important implementations.

Additional Context

This feature introduces reranking as a critical second-stage filter that re-scores retrieval results based on semantic relevance rather than just vector similarity. This dramatically improves the quality of context provided to the AI.

Related PR: #6609 implements this feature

[roomote]

Hi! I'm here to help out the maintainers and am going to see if I can fix the issue or at least get close enough to be helpful! I noticed there's already PR #6609 that implements this feature. I'll review it and help get it ready for merging.

[hannesrudolph]

Thanks for raising this. The discussion has been valuable but we haven't landed on a solution that makes sense yet.

Closing this to keep the issue tracker focused on actionable items. If you have a concrete proposal for improvement, please open a new issue using our templates and reference this thread. We're open to revisiting when there's a clearer path forward.

Appreciate the discussion and your patience while we figure out the right approach.