Discussion on SuperComponent abstraction

[julian-risch]

This is the discussion board for the experimental SuperComponent class.

This experiment allows to wrap any pipeline into a friendly component interface and to create your own super components.

🧑‍🍳 See the PR that adds the experiment here
🧑‍🍳 See an example of super components here

Note: Experimental features live in this repository for a fixed period of time. We don't guarantee that we will continue maintaining experimental features. But if they are successful, stable and if you like it, we will move the feature to the core Haystack package.

[d-kleine]

Would it be possible to also provide a full code example (in the docs or cookbook)?

As far as I understand it, it's a wrapper around single components. I would be particularly interested in knowing what the purpose of this SuperComponent is and how it is visualized in a pipeline via my_pipeline.show() (I haven't tested it yet, but I think it would be great if the included components were enclosed by a dashed line box representing the SuperComponent)

[julian-risch]

Hello @d-kleine here are two visualized pipelines, one with a MultiFileConverter component, which uses SuperComponent and one without. Currently, the MultiFileConverter hides all the complexity.

[julian-risch]

If you want to try out building with a SuperComponent, here is code to get started:

# pip install git+https://github.com/deepset-ai/haystack-experimental.git@multi-file-converter-unclassified
# pip install haystack-ai markdown-it-py mdit_plain openpyxl pandas pypdf python-docx python-pptx trafilatura


from haystack import Pipeline
from haystack.components.joiners import DocumentJoiner
from haystack.components.preprocessors import DocumentSplitter, DocumentCleaner
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore

from haystack_experimental.super_components.converters import MultiFileConverter

from pathlib import Path

document_store = InMemoryDocumentStore()
document_joiner = DocumentJoiner()

multi_file_converter = MultiFileConverter()

document_cleaner = DocumentCleaner()
document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50)
document_writer = DocumentWriter(document_store)

p = Pipeline()
p.add_component("converter", multi_file_converter)
p.add_component("cleaner", document_cleaner)
p.add_component("splitter", document_splitter)
p.add_component("writer", document_writer)
p.connect("converter", "cleaner")
p.connect("cleaner", "splitter")
p.connect("splitter", "writer")

p.show()

#p.run({"converter": {"sources": list(Path("YOUR_DIRECTORY").glob("**/*"))}})

[d-kleine]

Great point on this! We could definitely add a dedicated show method. But you should be able to do HybridRetriever.pipeline.show() to get what you want.

Yeah, that would be great! 👍🏻 🙂

Unfortunately, running HybridRetriever.pipeline.show() doesn't work for me after having defined the SC with @super_component:

AttributeError: type object 'HybridRetriever' has no attribute 'pipeline'

[sjrl]

Hmm good to know, I'll definitely take a look at that.

[sjrl]

Ahh okay I see the issue. You do need to initialize the Super Component before you can access the pipeline. So

from haystack.document_stores.in_memory.document_store import InMemoryDocumentStore

hybrid = HybridRetriever(InMemoryDocumentStore())

print(hybrid.pipeline)

should work

[d-kleine]

Yes, worked, thanks 👍🏻

hybrid.pipeline.show(server_url="http://localhost:3000") # running on Docker

Still a little confusing to be honest 😅

[d-kleine]

For your second point, I'd highly recommend checking out tracing or setting up logging.

I have checked this out, worked for the SC too. But tracing has been performed on component-level, and there is no information if a component belongs to a SC, which might be helpful to see in the logs (especially if the pipeline contains multiple components of the same kind, where one is used inside the SC and one outside of it)

[sjrl]

With the Haystack 2.13 release, we completed the SuperComponent experiment and here is the new docs page. And here is a tutorial on how to create your own.