Vision model issue with Azure OpenAI and serverless mode

[fdtmsft]

What happened?

When using Azure OpenAI in serverless mode and a custom endpoint, the query is sent to the backend URL without api-key in the headers.

This seems to be due to https://github.com/danny-avila/LibreChat/blob/main/api/app/clients/OpenAIClient.js#L1183 resetting the headers, if I instead merge the headers with this code opts.defaultHeaders = { ...opts.defaultHeaders, ...resolveHeaders(headers) }; the headers are correctly sent and the entire process is resolved. It also works if I remove this line altogether (but I am not familiar enough with the codebase to verify all potential side effects).

I can attach a PR with the above fix if bug is confirmed.

Version Information

ded3cd8

Steps to Reproduce

1. use serverless: true within an Azure OpenAI custom config (azureOpenAI/groups/serverless) as well as a custom baseURL and an apiKey
2. on LibreChat, upload an image, add a text
3. query is sent to the URL without the api-key header

What browsers are you seeing the problem on?

No response

Relevant log output

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Can't reproduce the issue. If using a serverless model, it falls back to a vision model if the current model does not support images.

In this case, I start with deepseek-r1, attach an image, and since it doesn't support images, it uses my gpt-4o config instead:

my config:

endpoints:
  azureOpenAI:
    streamRate: 35
    titleModel: "gpt-4o"
    titleConvo: true
    groups:
    - group: "eastus"
      apiKey: "${EASTUS_API_KEY}"
      instanceName: "my-eastus"
      version: "2024-02-15-preview"
      models:
        gpt-4o:
          deploymentName: gpt-4o
    - group: "region-eastus"
      apiKey: "${EASTUS2_API_KEY}"
      instanceName: "my-eastus2"
      version: "2024-12-01-preview"
      models:
        gpt-4o-latest:
          deploymentName: "gpt-4o-mini"
          version: "2024-02-15-preview"
        o1-mini:
          deploymentName: "o1-mini"
        o1-preview:
          deploymentName: "o1-preview"
        o3-mini:
          deploymentName: "o3-mini"
    - group: "region-sweden" # arbitrary name
      apiKey: "${SWEDEN_API_KEY}" # arbirary env. var, must be defined in .env file
      instanceName: "libre-dall-e" # the instance
      version: "2024-12-01-preview"
      models:
        o1-2024-12-17:
          deploymentName: "o1"
    - group: "Azure AI Foundry"
      apiKey: "${AZURE_DEEPSEEK_API_KEY}"
      baseURL: "https://DeepSeek-R1-foundry-endpoint.eastus.models.ai.azure.com/v1/"
      version: "2024-08-01-preview"
      serverless: true
      models:
        DeepSeek-R1:
          deploymentName: "DeepSeek-R1"

[fdtmsft]

It's not the model not supporting images, it's more the backend URL (which is not openAI directly but an API in front) does not receive the api-key (and returns a 401 error if this is not present). As I understand the spec, it should.

version: 1.2.1
endpoints:
    azureOpenAI:
        titleModel: "gpt-4o"
        assistants: false
        plugins: false
        summarize: true
        summaryModel: "gpt-4o"
        titleConvo: true
        groups:
          - group: "openai"
            assistants: false
            apiKey: xxxxxxxxxxxxxxxxxxxxxxxx
            baseURL: https://xxxxxxxxxxxxxxxxxxxxx/openai/deployments/gpt-4o
            serverless: true
            version: "2024-08-01-preview"
            models:
                gpt-4o:
                    deploymentName: gpt-4o

We are using the baseURL feature to reach an internal API (which supports OpenAI API queries) before going to AzureOpenAI.

You can verify the behavior by setting a breakpoint at query time, the headers are different if an image is included (base64 encoded) or not: api-key is not included. When updating the code as written in the initial post, the api-key is present and the model correctly receives the image as expected.

[danny-avila]

you should not be using serverless for OpenAI models

[fdtmsft]

Why not? It is documented as a viable option in the OpenAI section of the docs https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/azure_openai#serverless

Furthermore, beyond my specific settings it seems wrong to have a different header set sent to the backend API depending if text is sent or not (a header that is an api-key and not relevant to the text/text+image composition), whether serverless is the right setting in our case or not.