Add support for GPT-4 with Vision

[ghillert]

Having experimented with OpenAI's GPT-4 with Vision API, it would be amazing if Spring AI adds support for image-based input data (e.g. photos). This API allows you to post:

• One or more images
• Images are represented as base64 encoded data or as image URL

The API is located at: API reference: https://platform.openai.com/docs/guides/vision

An implementation could possibly also provide additional pre-processing in order to add support for video data as described here:

https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding

[markpollack]

Pull requests welcome.

[ghillert]

It looks like this feature would depend on TheoKanning/openai-java#397

[markpollack]

We have moved away from depending on that library and will provide our own implementations. ATM, brainstorming on having the API incorporate more than just chat models. Will be able to share a spike branch shortly. Was just circling back to issues filed wrt to other AI models. And happy new year Gunnar!

[arnokoehler]

How is the spike branch going? Is there a way others could help contribute?

[Mr-LiuDC]

Same issue.
Is there any progress?

[ThomasVitale]

It should be possible to close this issue since multimodality support has been introduced in Spring AI 1.0.0-M1, including the possibility to pass images as input to OpenAI: https://docs.spring.io/spring-ai/reference/api/multimodality.html

Example: https://github.com/ThomasVitale/llm-apps-java-spring-ai/tree/main/02-prompts/prompts-multimodality-openai

[markpollack]

Closing the issue. Thanks Thomas, issue pruning has been overdue, working through it now.