🖼️ Multimodal text generation

[anakin87]

Features

Here are the experimental features we've developed so far for multimodal text generation in Haystack. The experiment is ongoing, so expect more in the future.

• We introduced the ImageContent dataclass, which represents the image content of a user ChatMessage.
• The OpenAIChatGenerator was extended to support multimodal messages.
• We developed some image converter components.
• The ChatPromptBuilder was refactored to also work with string templates, making it easier to support multimodal use cases.

Notebook

For an introduction to the new features, check out the 📓 Introduction to Multimodal Text Generation notebook.
It also contains an application using textual retrieval + multimodal generation, and a multimodal Agent.

What's next?

You can follow the progress of the Multimodal experiment in this GitHub issue.
We'll also take care of keeping this discussion up to date.

In the future, you can expect support for more LLM providers, improvements to multimodal indexing and retrieval pipelines, plus the exploration of other interesting directions.

July updates

We introduced several new features, related to indexing and retrieval.
Check out #302 (comment).

Note: Experimental features live in this repository for a fixed period of time. We don't guarantee that we will continue maintaining experimental features. But if they are successful, stable and if you like it, we will move the feature to the core Haystack package.

[d-kleine]

Hi @anakin87 (and @julian-risch),

I just ran the cookbook notebook — great job on implementing the multimodal features! It’s clear a lot of thought went into making this both accessible and powerful!

I would like to share some of my personal thoughts and impressions:

Implementation

• I found the use cases in the notebook very clear and approachable, which made it easy for me to get started and see the value of the multimodality right away! Furthermore, I see a huge potential in the multimodal abilities, creating versatile applications that open up new possibilities for traditional NLP pipelines, especially in business scenarios!
• I’m a bit unsure about the necessity of converting images to a specific format for the pipeline. Personally, I’d love to know if I could just preprocess images using libraries I’m already comfortable with, like PIL or opencv-python. I have a feeling this is something others will ask about too, so some clarity here would be super helpful.
• With the ImageContent object, I found the separate from_url and from_file_path methods a little more complicated than I expected. If it’s possible to just have a single path or from_path method, that would make things feel a lot more intuitive (this might affect other components too).
• I have to admit, the Jinja templates still confuse me a bit. I’m not sure if it’s just a matter of getting more experience with them, but right now, they feel a bit unintuitive to me (just as a general thought)

Ideas/Requests

• One thing I was really curious about is how to handle different data types — like PDFs, images, and (XLSX/CSV) tables — all together in a pipeline. In my experience, real-world documents are rarely just one type, so seeing this in action would be super valuable for me.
• I’d also really appreciate some guidance on using local multimodal models, especially from HuggingFace (data protection and compliance are often a concern, so being able to run things locally is important)
• I’m also interested in seeing examples for processing audio and video files. Video, in particular, is fascinating to me because it combines both image and audio data, and I’d love to see how those could be analyzed together or separately.
• Finally, I think it would be awesome to have a full end-to-end pipeline example for a business use case — like a multimodal chatbot for medicine or marketing.

If you have any questions about my feedback or want to discuss any of these points further, please let me know. I’m genuinely excited to see where this goes! 🙂

[d-kleine]

I’ve also reviewed the sub-issues to understand the overall direction of the features, and I have a few observations to share:

• For PDF processing, it would be fantastic to have a Python package that works out of the box, without needing extra system installations. For example, I had to use WSL on Windows to run the code. This might be related to issue refactor: adopt pypdfium2 for PDF to image conversion #308.
• Regarding multimodal support, I really like @sjrl’s idea of a DocumentTypeRouter that uses the MIME type to choose how to handle different files.
• The DocumentLengthRouter is great for spotting empty pages, but sometimes “empty” pages actually have useful content, like page numbers or section titles. Maybe it would also useful to add a threshold setting to fine-tune what counts as “empty.”
• For PDFs, is it possible to process multiple pages at once using PDFToImageContent, specific pages? This would make handling larger documents much easier.
• I’m also curious about comparing information across different pages or even multiple documents — this seems like an exciting use case!
• Maybe a media type counter could also be handy, so users can quickly see what types of content they’re working with.
• Sometimes documents are scanned upside down or rotated. Would it be possible to add an Autorotater (either as a parameter or a separate tool) to automatically adjust page orientation for better processing?
• To enhance the multimodal capabilities, a local OCR converter could be a great addition. It could help save resources by separating text from images and handling them more efficiently.
• Lastly, it would be wonderful to support more languages — not just English, but also those using the Latin alphabet (like German and French), as well as other scripts (such as Arabic and Chinese). A multilingual retriever that finds relevant documents regardless of language would make the system even more powerful.

[sjrl]

Thanks for the feedback! I can quickly answer a few of your points:

• For PDF processing, it would be fantastic to have a Python package that works out of the box, without needing extra system installations. For example, I had to use WSL on Windows to run the code. This might be related to issue refactor: adopt pypdfium2 for PDF to image conversion #308.

Yup we totally agree! Once we release a new version of haystack-experimental you shouldn't need to install any extra system deps and instead directly use pypdfium2 via pip.

• Regarding multimodal support, I really like @sjrl’s idea of a DocumentTypeRouter that uses the MIME type to choose how to handle different files.

This has been merged and will be available via a new release of haystack-experimental.

• The DocumentLengthRouter is great for spotting empty pages, but sometimes “empty” pages actually have useful content, like page numbers or section titles. Maybe it would also useful to add a threshold setting to fine-tune what counts as “empty.”

Great idea! I'll keep that in mind as we work on this.

• For PDFs, is it possible to process multiple pages at once using PDFToImageContent, specific pages? This would make handling larger documents much easier.

Yes it does! It has a page_range init variable (can also be passed at runtime) which allows you to specifically pick pages or ranges of pages to process.

• Sometimes documents are scanned upside down or rotated. Would it be possible to add an Autorotater (either as a parameter or a separate tool) to automatically adjust page orientation for better processing?

This is an interesting idea. We can definitely take a look into this.

• To enhance the multimodal capabilities, a local OCR converter could be a great addition. It could help save resources by separating text from images and handling them more efficiently.

This is also something we can take a look into, but for now we will probably rely on our existing OCR converters like the AzureOCRConverter and first work on adding support for doing OCR with Vision LLMs.

• Lastly, it would be wonderful to support more languages — not just English, but also those using the Latin alphabet (like German and French), as well as other scripts (such as Arabic and Chinese). A multilingual retriever that finds relevant documents regardless of language would make the system even more powerful.

Totally agree! This isn't so much on us, but rather looking for models out there that do support multilingual retrieval. Unfortunately, for multilingual retreivers using image embeddings I don't think there are that many out there currently, but we are happy to hear any suggestions!

[d-kleine]

Thank you for your fast response!

Totally agree! This isn't so much on us, but rather looking for models out there that do support multilingual retrieval. Unfortunately, for multilingual retreivers using image embeddings I don't think there are that many out there currently, but we are happy to hear any suggestions!

Yeah, should be already possible with Haystack's Jina integration using the CLIP v2 model (with 89 supported languages): https://jina.ai/news/jina-clip-v2-multilingual-multimodal-embeddings-for-text-and-images/

Would be a great use case for the cookbook for multi-languages (and maybe even multimodal) pipelines imho 👍🏻

[anakin87]

UPDATE

We released v0.11.0, including several new multimodal features, mostly focused on indexing and retrieval.
A notebook will be published soon to show practical usage examples.

• multimodal support in AmazonBedrockChatGenerator
• new image Converters
• new Routers
• SentenceTransformersDocumentImageEmbedder: a component to compute embeddings for image-based documents
• LLMDocumentContentExtractor: a component to extract textual content from image-based documents using a vision-enabled LLM

[bilgeyucel]

All multimodality-related features are released with Haystack 2.16.0. Please use the discussion board from haystack for comments/suggestions 🙌