Discussion on EmbeddingBasedDocumentSplitter component

[davidsbatista]

This is the discussion board for EmbeddingBasedDocumentSplitter component.

We introduced the QueryExpander component, which generates a list of semantically similar queries to the user query to improve retrieval recall in RAG.

Usage example

from haystack import Document
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
from haystack_experimental.components.preprocessors import EmbeddingBasedDocumentSplitter
doc = Document(
    content="This is a first sentence. This is a second sentence. This is a third sentence. "
    "Completely different topic. The same completely different topic."
)
embedder = SentenceTransformersDocumentEmbedder()
splitter = EmbeddingBasedDocumentSplitter(
    document_embedder=embedder,
    sentences_per_group=2,
    percentile=0.95,
    min_length=50,
    max_length=1000
)
splitter.warm_up()
result = splitter.run(documents=[doc])

Try it out and let us know what you think in the comments 👇

Note: Experimental features live in this repository for a fixed period of time. We don't guarantee that we will continue maintaining experimental features. But if they are successful, stable and if you like it, we will move the feature to the core Haystack package.