Analyze image service call for Vision models.

[maxi1134]

Checklist

• I have filled out the template to the best of my ability.
• This only contains 1 feature request (if you have multiple feature requests, open one feature request for each feature request).
• This issue is not a duplicate feature request of previous feature requests.

Is your feature request related to a problem? Please describe.

I've been trying to get a vision model running with a rest_command.
It works sometimes, and am wondering if this integration could add a service call for that, just like OpenAI has in HA.

Describe the solution you'd like

An action to analyze an image ( URL or Entity in HA)

Describe alternatives you've considered

A curl rest command.

Nominally:

    url: http://192.168.0.14:3000/api/chat/completions
    method: POST
    content_type: 'application/json'
    timeout: 120
    headers:
      authorization: !secret open_webui_rest_headers_secret
    payload: > 
      {
            "messages": [
                {
                    "role": "image_url",
                    "content": "{{url}}",
                    "temperature": 0.1
                },
                {
                    "role": "user",
                    "content": "{{prompt}}",
                    "temperature": 0.1
                }
            ],
            "model": "maxi_llama32-vision:latest"
        }

Additional context

It would be great if the prompt is modular as well.

[TheRealPSV]

Is this a feature available in OpenWebUI itself? If so, can you give me a link to it in their docs or a screenshot of how it's used?

[maxi1134]

Is this a feature available in OpenWebUI itself? If so, can you give me a link to it in their docs or a screenshot of how it's used?

It is available but can only be piped in a BASE64 encoded Image, which is my roadblock using a REST_command entity.

The callback is

{ "messages": [ { "role": "user", "content": [ { "type": "text", "text": "is there a woman on this image?" }, { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64VALUE" } } ] } ], "model": "maxi_llama32-vision:latest" }

[TheRealPSV]

It is available but can only be piped in a BASE64 encoded Image

I guess my next question then is, how would you pass the image into Home Assistant? As far as I'm aware, the Conversation Agent will only accept text inputs.

[maxi1134]

I guess my next question then is, how would you pass the image into Home Assistant? As far as I'm aware, the Conversation Agent will only accept text inputs.

Either a Camera entity or a /local/www/* URL would work for me.

[TheRealPSV]

I'm not sure this makes sense for the Conversation Agent, it sounds like it would be better suited to an action, or something like that. Happy to accept a pull request if someone wants to do the heavy lifting, but I don't think I can add what you're looking for, at least not right now.