feat: add more advanced fusion strategies (#492)

Author: ludwiktrammer

docs/api_reference/core/hybrid.md

@@ -0,0 +1,9 @@
+# Hybrid Vector Store & Fusion Strategies
+
+::: ragbits.core.vector_stores.hybrid.HybridSearchVectorStore
+
+::: ragbits.core.vector_stores.hybrid_strategies.OrderedHybridRetrivalStrategy
+
+::: ragbits.core.vector_stores.hybrid_strategies.ReciprocalRankFusion
+
+::: ragbits.core.vector_stores.hybrid_strategies.DistributionBasedScoreFusion

docs/api_reference/core/vector-stores.md

@@ -6,8 +6,6 @@
 
 ::: ragbits.core.vector_stores.base.VectorStore
 
-::: ragbits.core.vector_stores.hybrid.HybridSearchVectorStore
-
 ::: ragbits.core.vector_stores.in_memory.InMemoryVectorStore
 
 ::: ragbits.core.vector_stores.chroma.ChromaVectorStore

docs/how-to/document_search/search-documents.md

@@ -22,6 +22,28 @@ Searching for elements is performed using a vector store. [`DocumentSearch`][rag
 
     One of the simplest vector search strategies used in Ragbits is dense search. This approach leverages an embedding model to generate vector representations of search queries and compares them against the dense vector representations of ingested elements. It is a straightforward method and often serves as a good starting point for developing a retrieval pipeline.
 
+=== "Hybrid search"
+
+    ```python
+    from ragbits.core.embeddings import LiteLLMEmbedder
+    from ragbits.core.vector_stores.qdrant import QdrantVectorStore
+    from ragbits.core.vector_stores.hybrid import HybridSearchVectorStore
+    from ragbits.document_search import DocumentSearch
+
+    embedder = LiteLLMEmbedder(model="text-embedding-3-small", ...)
+    vector_store_text = InMemoryVectorStore(embedder=embedder, index_name="text_index", embedding_type=EmbeddingType.TEXT)
+    vector_store_image = InMemoryVectorStore(embedder=embedder, index_name="image_index", embedding_type=EmbeddingType.IMAGE)
+    vector_store = HybridSearchVectorStore(vector_store_text, vector_store_image)
+
+    document_search = DocumentSearch(vector_store=vector_store, ...)
+
+    elements = await document_search.search("What is the capital of Poland?")
+    ```
+
+    Hybrid search is a more advanced strategy that combines multiple vector stores, each optimized for different types of data or embedding models. This approach allows for more flexible and efficient retrieval, as it can leverage the strengths of different vector stores to improve search results. For example, you can combine dense and sparse vector stores or use different embedding models for different data types, or like in this example, use one store for text embeddings and another for image embeddings of the same entry.
+
+    To learn more about using Hybrid Search, refer to [How to Perform Hybrid Search with Multiple Vector Stores](../vector_stores/hybrid.md).
+
 ## Rephrase query
 
 By default, the input query is provided directly to the embedding model. However, there is an option to add an additional step before vector search. Ragbits offers several common rephrasing techniques that can be utilized to refine the query and generate better embeddings for retrieval.

docs/how-to/prompts/prompts_lab.md

@@ -3,7 +3,7 @@
 Prompts Lab is a GUI tool that automatically detects prompts in your project and allows you to interact with them. You can use it to test your prompts with Large Language Models and see how the model responds to different prompts.
 
 !!! note
-    To follow this guide, ensure that you have installed the `ragbits` package and are in a directory with Python files that define some ragbits prompts (usually, this would be the root directory of your project) in your command line terminal. If you haven't defined any prompts yet, you can use the `SongPrompt` example from [Ragbit's Quickstart Guide](../../quickstart/quickstart1_prompts.md) and save it in a Python file with a name starting with "prompt_" in your project directory.
+    To follow this guide, ensure that you have installed the `ragbits` package and are in a directory with Python files that define some ragbits prompts (usually, this would be the root directory of your project) in your command line terminal. If you haven't defined any prompts yet, you can use the `SongPrompt` example from [Ragbits' Quickstart Guide](../../quickstart/quickstart1_prompts.md) and save it in a Python file with a name starting with "prompt_" in your project directory.
 
 ## Starting Prompts Lab
 
@@ -35,7 +35,7 @@ Then, click "Render prompt" to view the final prompt content, with all placehold
 !!! note
     If there is no [preferred LLM configured for your project](../project/component_preferences.md), Prompts Lab will use OpenAI's gpt-3.5-turbo. Ensure that the OPENAI_API_KEY environment variable is set and contains your OpenAI API key.
 
-    Alternatively, you can use your own custom LLM factory (a function that creates an instance of [ragbit's LLM class][ragbits.core.llms.LLM]) by specifying the path to the factory function using the `--llm-factory` option with the `ragbits prompts lab` command.
+    Alternatively, you can use your own custom LLM factory (a function that creates an instance of [Ragbits' LLM class][ragbits.core.llms.LLM]) by specifying the path to the factory function using the `--llm-factory` option with the `ragbits prompts lab` command.
 
 
 ## Conclusion

docs/how-to/vector_stores/hybrid.md

@@ -0,0 +1,117 @@
+# How to Perform Hybrid Search with Multiple Vector Stores
+
+Ragbits comes with a special type of vector store called [`HybridSearchVectorStore`][ragbits.core.vector_stores.hybrid.HybridSearchVectorStore], which allows you to combine multiple vector stores into a single search index. It acts as a single vector store but internally manages querying and updating multiple vector stores during operations like storing, searching, and deleting entries.
+
+The main use cases for using a hybrid vector store are:
+
+* **Combining Different Modalities**: You can combine multiple vector stores that store different types of data, like text and images. This allows you to store multiple modality-specific vectors for the same entry (for example, an image embedding and a text embedding of a description of the image) and search them together.
+* **Combining Different Types of Embeddings**: You can combine multiple vector stores that store different types of embeddings, like dense and sparse embeddings. This allows you to store multiple embeddings for the same entry and search them simultaneously.
+
+!!! info
+    <!-- TODO: Remove this once sparse embedding support in Vector Stores is implemented -->
+    Sparse embeddings support in Vector Stores is an upcoming feature of Ragbits. The examples below will be updated to show how to use them with hybrid search once they are available.
+
+## Using a Hybrid Vector Store with Different Modalities
+
+To create a hybrid vector store, you need to pass a list of vector stores to the constructor of the [`HybridSearchVectorStore`][ragbits.core.vector_stores.hybrid.HybridSearchVectorStore] class. For example, this creates two in-memory vector stores—one for text and one for images:
+
+```python
+from ragbits.core.vector_stores.hybrid import HybridSearchVectorStore
+from ragbits.core.vector_stores.in_memory import InMemoryVectorStore
+from ragbits.core.embeddings.vertex_multimodal import VertexAIMultimodelEmbedder
+
+embedder = VertexAIMultimodelEmbedder()
+
+vector_store_text = InMemoryVectorStore(embedder=embedder, embedding_type=EmbeddingType.TEXT)
+vector_store_image = InMemoryVectorStore(embedder=embedder, embedding_type=EmbeddingType.IMAGE)
+
+vector_store_hybrid = HybridSearchVectorStore(vector_store_text, vector_store_image)
+```
+
+You can then use the `vector_store_hybrid` object to store, search, and delete entries, just as you would use a regular vector store, or pass it to [Ragbits' Document Search](../document_search/ingest-documents.md). When you store an entry in the hybrid vector store, it will be stored in all the vector stores it contains. In this case, one will store the text embedding and the other will store the image embedding.
+
+## Using a Hybrid Vector Store with Different Types of Embeddings
+
+<!-- TODO: Change this example to dense and sparse embeddings once sparse embedding support in Vector Stores is implemented -->
+Similarly, you can create a hybrid vector store with different types of embeddings. For example, this creates two in-memory vector stores—one using an embedding model from OpenAI and one using an embedding model from Mistral:
+
+```python
+from ragbits.core.vector_stores.hybrid import HybridSearchVectorStore
+from ragbits.core.vector_stores.in_memory import InMemoryVectorStore
+from ragbits.core.embeddings.litellm import LiteLLMEmbedder
+
+vector_store_openai = InMemoryVectorStore(embedder=LiteLLMEmbedder(model="text-embedding-ada-002"))
+vector_store_mistral = InMemoryVectorStore(embedder=LiteLLMEmbedder(model="mistral/mistral-embed"))
+
+vector_store_hybrid = HybridSearchVectorStore(vector_store_openai, vector_store_mistral)
+```
+
+You can then use the `vector_store_hybrid` object to store, search, and delete entries, just as you would use a regular vector store, or pass it to [Ragbits' Document Search](../document_search/ingest-documents.md). When you store an entry in the hybrid vector store, it will be stored in all the vector stores it contains. In this case, one will store the embedding using the OpenAI model and the other will store the embedding using the Mistral model.
+
+Note that you can pass an arbitrary number of vector stores to the `HybridSearchVectorStore` constructor, and they can be of any type as long as they implement the `VectorStore` interface. For example, this combines three vector stores—one Chroma vector store, one Qdrant vector store, and one PgVector vector store:
+
+```python
+import asyncpg
+from chromadb import EphemeralClient
+from qdrant_client import AsyncQdrantClient
+
+from ragbits.core.vector_stores.hybrid import HybridSearchVectorStore
+from ragbits.core.vector_stores.chroma import ChromaVectorStore
+from ragbits.core.vector_stores.qdrant import QdrantVectorStore
+from ragbits.core.vector_stores.pgvector import PgVectorStore
+from ragbits.core.embeddings.litellm import LiteLLMEmbedder
+
+postgres_pool = await asyncpg.create_pool("postgresql://user:password@localhost/db")
+
+vector_store_hybrid = HybridSearchVectorStore(
+    ChromaVectorStore(
+        client=EphemeralClient(),
+        index_name="chroma_example",
+        embedder=LiteLLMEmbedder(),
+    ),
+    QdrantVectorStore(
+        client=AsyncQdrantClient(location=":memory:"),
+        index_name="qdrant_example",
+        embedder=LiteLLMEmbedder(),
+    ),
+    PgVectorStore(
+        client=pool,
+        table_name="postgres_example",
+        vector_size=1536,
+        embedder=LiteLLMEmbedder(),
+    ),
+)
+
+# The entry will be stored in all three vector stores
+await vector_store_hybrid.store([VectorStoreEntry(id=uuid.uuid4(), text="Example entry")])
+```
+
+## Specifying the Retrieval Strategy for a Hybrid Vector Store
+
+When you search a hybrid vector store, you can specify a retrieval strategy to determine how the results from the different vector stores are combined. Ragbits comes with the following retrieval strategies:
+
+* [`OrderedHybridRetrivalStrategy`][ragbits.core.vector_stores.hybrid_strategies.OrderedHybridRetrivalStrategy]: This strategy returns the results from the vector stores ordered by their score. If the same entry is found in multiple vector stores, either the highest score is used or if the `sum_scores` parameter is set to `True`, the scores are summed. This is the default strategy.
+* [`ReciprocalRankFusion`][ragbits.core.vector_stores.hybrid_strategies.ReciprocalRankFusion]: This strategy combines the results from the vector stores using the [Reciprocal Rank Fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) algorithm, which prioritizes entries that appear at the top of the results from individual vector stores. If the same entry is found in multiple vector stores, the scores are summed by default, or if the `sum_scores` parameter is set to `False`, the highest score is used.
+* [`DistributionBasedScoreFusion`][ragbits.core.vector_stores.hybrid_strategies.DistributionBasedScoreFusion]: This strategy combines the results from the vector stores using the [Distribution-Based Score Fusion](https://medium.com/plain-simple-software/distribution-based-score-fusion-dbsf-a-new-approach-to-vector-search-ranking-f87c37488b18) algorithm, which normalizes the scores from the individual vector stores so they can be compared and combined sensibly. If the same entry is found in multiple vector stores, either the highest score is used or if the `sum_scores` parameter is set to `True`, the scores are summed.
+
+Note that summing the scores from individual stores boosts the entries found in multiple stores. This can be useful when searching through multiple types of embeddings but may not be desirable when searching through multiple modalities since entries containing both text and image embeddings would have an advantage over those containing only one.
+
+To specify a retrieval strategy when searching a hybrid vector store, you can pass it as the `retrieval_strategy` parameter to the constructor of the [`HybridSearchVectorStore`][ragbits.core.vector_stores.hybrid.HybridSearchVectorStore] class. For example, this creates a hybrid vector store with the `DistributionBasedScoreFusion` retrieval strategy:
+
+```python
+from ragbits.core.vector_stores.hybrid import HybridSearchVectorStore
+from ragbits.core.vector_stores.in_memory import InMemoryVectorStore
+from ragbits.core.vector_stores.hybrid_strategies import DistributionBasedScoreFusion
+from ragbits.core.embeddings.litellm import LiteLLMEmbedder
+
+embedder = LiteLLMEmbedder()
+
+vector_store_text = InMemoryVectorStore(embedder=embedder, embedding_type=EmbeddingType.TEXT)
+vector_store_image = InMemoryVectorStore(embedder=embedder, embedding_type=EmbeddingType.IMAGE)
+
+vector_store_hybrid = HybridSearchVectorStore(
+    vector_store_text,
+    vector_store_image,
+    retrieval_strategy=DistributionBasedScoreFusion(),
+)
+```

docs/quickstart/quickstart2_rag.md

@@ -1,6 +1,6 @@
 # Quickstart 2: Adding RAG Capabilities
 
-In this chapter, we will explore how to use Ragbit's Document Search capabilities to retrieve relevant documents for your prompts. This technique is based on the Retrieval Augmented Generation (RAG) architecture, which allows the LLM to generate responses informed by relevant information from your documents.
+In this chapter, we will explore how to use Ragbits' Document Search capabilities to retrieve relevant documents for your prompts. This technique is based on the Retrieval Augmented Generation (RAG) architecture, which allows the LLM to generate responses informed by relevant information from your documents.
 
 To work with document content, we first need to "ingest" them (i.e., process, embed, and store them in a vector database). Afterwards, we can search for relevant documents based on the user's input and use the retrieved information to enhance the LLM's response.
 

mkdocs.yml

@@ -19,6 +19,7 @@ nav:
         - "Interact with LLMs": how-to/llms/use_llms.md
         - "Use local or self-hosted LLMs": how-to/llms/use_local_llms.md
       - Vector Stores:
+        - "Perform hybrid search": how-to/vector_stores/hybrid.md
         - "Use PostgreSQL as a vector store with pgvector": how-to/vector_stores/use_pgVector_store.md
       - Project configuration:
         - "Set preferred components in project": how-to/project/component_preferences.md
@@ -43,6 +44,7 @@ nav:
           - api_reference/core/llms.md
           - api_reference/core/embeddings.md
           - api_reference/core/vector-stores.md
+          - api_reference/core/hybrid.md
       - Document Search:
           - api_reference/document_search/index.md
           - api_reference/document_search/documents.md

packages/ragbits-core/CHANGELOG.md

@@ -1,6 +1,7 @@
 # CHANGELOG
 
 ## Unreleased
+- Add new fusion strategies for the hybrid vector store: RRF and DBSF (#413)
 
 ## 0.13.0 (2025-04-02)
 - Make the score in VectorStoreResult consistent (always bigger is better)

packages/ragbits-core/src/ragbits/core/vector_stores/base.py

@@ -55,6 +55,9 @@ class VectorStoreResult(BaseModel):
     vector: list[float]
     score: float
 
+    # If the results were created by combining multiple results, this field will contain the subresults.
+    subresults: list["VectorStoreResult"] = []
+
 
 class VectorStoreOptions(Options):
     """

packages/ragbits-core/src/ragbits/core/vector_stores/hybrid.py

@@ -1,4 +1,3 @@
-import abc
 import asyncio
 from uuid import UUID
 
@@ -10,46 +9,7 @@
     VectorStoreResult,
     WhereQuery,
 )
-
-
-class HybridRetrivalStrategy(abc.ABC):
-    """
-    A class that can join vectors retrieved from different vector stores into a single list,
-    allowing for different strategies for combining results.
-    """
-
-    @abc.abstractmethod
-    def join(self, results: list[list[VectorStoreResult]]) -> list[VectorStoreResult]:
-        """
-        Joins the multiple lists of results into a single list.
-
-        Args:
-            results: The lists of results to join.
-
-        Returns:
-            The joined list of results.
-        """
-
-
-class OrderedHybridRetrivalStrategy(HybridRetrivalStrategy):
-    """
-    A class that orders the results by score and deduplicates them by choosing the first occurrence of each entry.
-    """
-
-    def join(self, results: list[list[VectorStoreResult]]) -> list[VectorStoreResult]:  # noqa: PLR6301
-        """
-        Joins the multiple lists of results into a single list.
-
-        Args:
-            results: The lists of results to join.
-
-        Returns:
-            The joined list of results.
-        """
-        all_results = [result for sublist in results for result in sublist]
-        all_results.sort(key=lambda result: result.score, reverse=True)
-
-        return list({result.entry.id: result for result in all_results}.values())
+from ragbits.core.vector_stores.hybrid_strategies import HybridRetrivalStrategy, OrderedHybridRetrivalStrategy
 
 
 class HybridSearchVectorStore(VectorStore):