how is image input used for multi-modal models?

[weipienlee]

Using o3-mini model to describe an image surprisingly works at the LibreChat's Azure endpoint while the model officially doesn't support it yet. Test with langchain confirms the lack of support (see code and message below). What actually happens in LibreChat with the input image?

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "Your a friendly AI assistant."),
        (
            "user",
            [
                {
                    "type": "text",
                    "text": "{query}"
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "data:image/jpeg;base64,{image_data}"},
                }
            ],
        ),
    ]
)

error message:
'Invalid content type. image_url is only supported by certain models.'

update: i'm on 0.7.7

[weipienlee]

I narrowed it down a bit. It seems that if at least 1 other model supports it that makes the non-supporting one seem to support input image. i.e. if only the non-supported model is listed e.g. "o3-mini-2025-01-31" than it indeed fails to accept an image as input.

  azureOpenAI:
    titleModel: "BM-gpt-4o-mini"
    titleConvo: true 
    plugins: false
    assistants: true
    groups:
    - group: "xxxxxxx" # arbitrary name
      apiKey: "${AZURE_OPENAI_API_KEY_xxxxxx}" # Azure OpenAI API key
      instanceName: "${AZURE_OPENAI_INSTANCE_xxxxxx}" # name of the resource group or instance
      plugins: false
      assistants: true
      version: "${AZURE_OPENAI_API_VERSION_xxxxxxx}"
      models:
        BM-o3-mini-2025-01-31: 
          deploymentName: o3-mini-2025-01-31
        BM-gpt-4o:
          deploymentName: gpt-4o

error message when only "o3-mini-2025-01-31" is listed:

2025-03-28 12:17:13 2025-03-28 11:17:13 warn: [OpenAIClient.chatCompletion][stream] API error
2025-03-28 12:17:13 2025-03-28 11:17:13 error: 
2025-03-28 12:17:13 2025-03-28 11:17:13 error: [handleAbortError] AI response error; aborting request: 400 Invalid content type. image_url is only supported by certain models.

so that makes sense now