Support for rerank and colbert

[rjmalagon]

Rerank models are very useful to empower RAG, help a lot with search on RAG and they are resource intensive. It would be very nice to accelerate rerank via llama.cpp, to make it accessible just like embedding.

Colbert models are a more complex tool, between rerank and embedding, but at the end, just an optimized alternative to rerank, very welcome if supported by llama.cpp too.

Actual implementations are strictly transformers based.

https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1
https://huggingface.co/mixedbread-ai/mxbai-colbert-large-v1

This could allow Open-webui to offload this to Ollama. (open-webui+ollama , maybe the most accessible tools for local RAG)

[ciekawy]

I did open feature request for reranking #8555

[onestardao]

Rerankers like ColBERT are definitely powerful — but from what I’ve seen in local RAG setups, they often introduce subtle issues that don’t show up right away.

Some of the recurring failure patterns I’ve encountered:

• No.5 Semantic ≠ Embedding — rerankers optimize on vector similarity, which doesn’t always track actual user intent or logic
• No.9 Entropy Collapse — without diversity boosting, many setups collapse to a few repetitive patterns or degenerate rankings
• No.6 Logic Collapse — even when the rerank is “correct”, it sometimes breaks downstream multi-step reasoning

These don’t mean rerankers are bad — just that they need careful semantic shaping or fallback strategies to avoid regressions.

Curious if others here have run into similar reranker pathologies? I’ve been experimenting with some layer-level entropy control to stabilize rankings — happy to share if helpful.