[Feature]: Add vision/multimodal model type for image, video, and pdf analysis

[kevincolten]

Feature Description

Add a new vision model type to esperanto that supports multimodal LLMs capable of processing images, videos, and pdfs. This would enable:

1. A create_vision() factory method (or extend create_language() with vision capability detection)
2. Image input support via base64-encoded images in message content
3. Provider-specific handling for vision-capable models (GPT-4o, Claude 3, Gemini Pro Vision, LLaVA, etc.)

---

Why would this be helpful?

Vision-capable LLMs are increasingly common, but esperanto currently only supports text-based language models. Adding vision support would enable:

• PDF analysis - Render pages as images for visual understanding (tables, charts, layouts)
• Video processing - Extract frames and analyze visual content
• Image-heavy documents - Process scanned documents, diagrams, screenshots
• Multi-provider flexibility - Use local models (LLaVA via Ollama) or cloud APIs (GPT-4o, Claude 3, Gemini)

This aligns with esperanto's goal of providing a unified interface across AI providers.

---

A proof-of-concept implementation exists in a closed PR: lfnovo/open-notebook#533

That implementation includes:

• default_vision_model configuration field
• get_vision_model() method with fallback to chat model
• Utilities for image encoding, video frame extraction, and PDF page rendering

The vision model provisioning works with the existing LangChain .to_langchain() interface - the main gap is esperanto's model registry and capability detection.

---

Contribution

• I am a developer and would like to work on implementing this feature (pending maintainer approval)

[lfnovo]

I am still questioning whether we should implement a new method like 'create_vision` or if we should just support Base64 in the create_language version (so we could expand it to audio and video later). That would be an even simpler implementation. What do you think?

[leejangthegreat]

I agree that adding vision capability is important for current version of esperanto, not only for vision-processing endpoints, but for newest chat models as well.

---

I wonder what changes is required for a comprehensive framework. Can we conclude this demand as:

1. Add a rpovider type for "vision" processing (or two for image / video separately). This type of models post request to image / video processing endpoints of those platforms ( https://api.openai.com/v1/images/xxx for OpenAI ) and get process results in specific format;
2. Add capability of handling files / images inputs ("file" in general) to LanguageModel in esperanto so that we can fully utilize those cutting-edge LLMs. Depending on different API platforms, we may either upload files to specific endpoints in advance, or directly encode files as base64 in request payloads ?

[leejangthegreat]

I am still questioning whether we should implement a new method like 'create_vision` or if we should just support Base64 in the create_language version (so we could expand it to audio and video later). That would be an even simpler implementation. What do you think?

Well, for some platforms the specific "create_vision" type is useful I think. There are some models accepting vision inputs, editing it by given prompts, and return vision results. #78 also mentioned this kind of payload.

[kevincolten]

I agree that extending create_language() with base64 content-array seems cleanest and saves the create_vision method name for future vision response features. It's really just a message format expansion, most major providers already support this through their chat completions endpoint as far as I know.

For the application-level features from lfnovo/open-notebook#533 (FFmpeg frame extraction, PDF rendering, LibreOffice conversion) would content-core be the better place for those? It already handles media extraction.

Let me know if I should start on the esperanto PR.