Impossible to upload files when using assistants into Azure OpenAI, neither in the knowledge base, nor in the chat session

[djuillard]

Hi,

I setup Librechat with docker-compose. I first used openai API directly, but for security purpose, I now want to swtich to Azure Openai (and in the end remove openai endpoint to only use Azure Openai).

I managed to setup openai endpoint and make it work for chat completions, and also for Assistants (both on azure openai).

For chat completion, I set up the environment variables to make it use azure openai embeddings through the RAG_API service, and it works fine : I can load a file in the chat session, and the RAG mecanism works well.

But when it comes to azure assistants, chat conversation works (and the assistant follows the instructions I have written), but I cannot upload any file, neither into the "knowledge" part of the assistant builder, nor in the chat windows, even though these 2 errors seem differents.

1. when I try load a file in the upload section of the assistant builder (as described in the screenshot below), I get the following error in the logs : LibreChat-API | 2024-06-19 13:25:12 error: [/files] Error processing file: 404 Resource not found
   

2. when I try to upload a file in the chat window of the assistant mode, I get an error directly in the chat window saying that "the Assistant run failed to initialize. Try sending a message in a new conversation". And I also got a more detailed error message in the logs : LibreChat-API | 2024-06-19 13:28:38 error: [/assistants/chat/] 400 Unknown parameter: 'file_ids'.
   

Here is how I configured my .env file (for security purpose, I removed the secrets of course) :

#===================================================#
#                     Endpoints                     #
#===================================================#

ENDPOINTS=openAI,assistants,anthropic,azureOpenAI,azureAssistants


#===================================#
# Known Endpoints - librechat.yaml  #
#===================================#
# https://docs.librechat.ai/install/configuration/ai_endpoints.html

GROQ_API_KEY=my_groq_key
# SHUTTLEAI_KEY=
OPENROUTER_KEY=my_openrouter_key
MISTRAL_API_KEY=my_mistral_key
# ANYSCALE_API_KEY=
# FIREWORKS_API_KEY=
# PERPLEXITY_API_KEY=
# TOGETHERAI_API_KEY=
OVHCLOUD_API_KEY=my_ovh_key
AZUREOPENAI_FRANCE_API_KEY=my_azure_openai_key_for_french_region
AZUREOPENAI_SWEDEN_API_KEY=my_azure_openai_key_for_swedish_region


#============#
# Azure      #
#============#


RAG_AZURE_OPENAI_ENDPOINT=https://azureopenaionbehalfaiinstance2sweden.openai.azure.com/
#DEBUG_RAG_API=true
RAG_AZURE_OPENAI_API_KEY=same_azure_openai_key_for_swedish_region_as_above
#RAG_AZURE_OPENAI_API_VERSION=2024-05-01-preview
EMBEDDINGS_PROVIDER=azure
EMBEDDINGS_MODEL=text-embedding-3-large

# Note: these variables are DEPRECATED
# Use the `librechat.yaml` configuration for `azureOpenAI` instead
# You may also continue to use them if you opt out of using the `librechat.yaml` configuration

# AZURE_OPENAI_DEFAULT_MODEL=gpt-3.5-turbo # Deprecated
# AZURE_OPENAI_MODELS=gpt-3.5-turbo,gpt-4 # Deprecated
# AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE # Deprecated
# AZURE_API_KEY= # Deprecated
# AZURE_OPENAI_API_INSTANCE_NAME= # Deprecated
# AZURE_OPENAI_API_DEPLOYMENT_NAME= # Deprecated
# AZURE_OPENAI_API_VERSION= # Deprecated
# AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME= # Deprecated
# AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME= # Deprecated
# PLUGINS_USE_AZURE="true" # Deprecated



#============#
# OpenAI     #
#============#

OPENAI_API_KEY=my_openai_key
# OPENAI_MODELS=gpt-3.5-turbo-0125,gpt-3.5-turbo-0301,gpt-3.5-turbo,gpt-4,gpt-4-0613,gpt-4-vision-preview,gpt-3.5-turbo-0613,gpt-3.5-turbo-16k-0613,gpt-4-0125-preview,gpt-4-turbo-preview,gpt-4-1106-preview,gpt-3.5-turbo-1106,gpt-3.5-turbo-instruct,gpt-3.5-turbo-instruct-0914,gpt-3.5-turbo-16k

DEBUG_OPENAI=false


#====================#
#   Assistants API   #
#====================#

ASSISTANTS_API_KEY=my_openai_assistant_key
ASSISTANTS_BASE_URL=
ASSISTANTS_MODELS=gpt-3.5-turbo-0125,gpt-3.5-turbo-16k-0613,gpt-3.5-turbo-16k,gpt-3.5-turbo,gpt-4,gpt-4-0314,gpt-4-32k-0314,gpt-4-0613,gpt-3.5-turbo-0613,gpt-3.5-turbo-1106,gpt-4-0125-preview,gpt-4-turbo-preview,gpt-4-1106-preview

...

And here is how I configured my librechat.yaml file :

# Configuration version (required)
version: 1.0.9

# Cache settings: Set to true to enable caching
cache: true

# Custom interface configuration
interface:
  # Privacy policy settings
  privacyPolicy:
    externalUrl: 'https://librechat.ai/privacy-policy'
    openNewTab: true

  # Terms of service
  termsOfService:
    externalUrl: 'https://librechat.ai/tos'
    openNewTab: true

# Example Registration Object Structure (optional)
registration:
  socialLogins: ['github', 'google', 'discord', 'openid', 'facebook']
  # allowedDomains:
  # - "gmail.com"

# rateLimits:
#   fileUploads:
#     ipMax: 100
#     ipWindowInMinutes: 60  # Rate limit window for file uploads per IP
#     userMax: 50
#     userWindowInMinutes: 60  # Rate limit window for file uploads per user
#   conversationsImport:
#     ipMax: 100
#     ipWindowInMinutes: 60  # Rate limit window for conversation imports per IP
#     userMax: 50
#     userWindowInMinutes: 60  # Rate limit window for conversation imports per user

# Definition of custom endpoints
endpoints:
  azureAssistants:
     disableBuilder: false # Disable Assistants Builder Interface by setting to `true`
     pollIntervalMs: 750  # Polling interval for checking assistant updates
     timeoutMs: 180000  # Timeout for assistant operations
  #   # Should only be one or the other, either `supportedIds` or `excludedIds`
  #   supportedIds: ["asst_supportedAssistantId1", "asst_supportedAssistantId2"]
  #   # excludedIds: ["asst_excludedAssistantId"]
  #   # (optional) Models that support retrieval, will default to latest known OpenAI models that support the feature
     retrievalModels: ["gpt-4"]
  #   # (optional) Assistant Capabilities available to all users. Omit the ones you wish to exclude. Defaults to list below.
     capabilities: ["code_interpreter", "actions", "tools"]
     privateAssistants: true

  custom:
    ...

  azureOpenAI:
    titleModel: "gpt-35-turbo"
    plugins: true
    assistants: true
    groups:
      - group: "Azure_OpenAI_RG1"
        apiKey: "${AZUREOPENAI_SWEDEN_API_KEY}"
        instanceName: "AzureOpenAIOnBehalfAIInstance2Sweden"
        deploymentName: "GPT-4-SWEDEN"
        assistants: true
        version: "2024-05-01-preview" 
      #  additionalHeaders:
      #    CustomHeader: "HeaderValue"
      #  addParams:
      #    max_tokens: 4096
      #    temperature: 0.7
      #  dropParams:
      #    - "frequency_penalty"
      #    - "presence_penalty"
        models:
          gpt-4:
            deploymentName: "GPT-4-SWEDEN"
            version: "2024-05-01-preview"
          gpt-35-turbo: 
            deploymentName: "GPT-35-turbo"
            version: "2024-05-01-preview"
          gpt-4o: 
            deploymentName: "GPT-4o"
            version: "2024-05-01-preview"
          dall-e-3: 
            deploymentName: "Dall-E3"
            version: "2024-05-01-preview"
          text-embedding-ada-002: 
            deploymentName: "embedding-ada-002-sweden"
            version: "2024-05-01-preview"

[danny-avila]

This is because Assistants through Azure are still using their "v1" implementation, where retrieval/file search was not possible

I started work on migrating it to v2 already, and will implement it soon: #2820

[djuillard]

OK, thanks for your (very!) quick reply. Glad to hear that this enhancement is already on track !

Great overall job by the way, by far the best opensource GenAI chatbot (and congrats as well for the documentation, because this is key !).

[danny-avila]

thanks so much!

[doubianimehdi]

Where are we on that request ? because my company wants to use librechat for custom assistants and it's the only real issue we have , being able to load files and store the embeddings for use in custom gpts with azure ai search is important ... can update us on that ? if you can tell us what it would need to make this issue go forward ? thank you !

[danny-avila]

Check out Agents, it's already available to use with Azure

https://www.librechat.ai/docs/features/agents