Need help to make Dall-e3 plugin work on Azure OpenAI

[djuillard]

Hi,

I have carefully read the documentation about plugins, and especially the section related to Azure, but I cannot make the Dall-E-3 plugin work so far.

When I try to activate it, here is what I get :

I do not see Dall-E-3 in the list, and I cannot neither click on the plugin store (nothing happens).

Here is my LibreChat.yaml conf :

#=====================================================================#
#                       LibreChat Configuration                       #
#=====================================================================#
# Please refer to the reference documentation for assistance          #
# with configuring your LibreChat environment. The guide is           #
# available both online and within your local LibreChat               #
# directory:                                                          #
# Online: https://docs.librechat.ai/install/configuration/dotenv.html #
# Locally: ./docs/install/configuration/dotenv.md                     #
#=====================================================================#

#==================================================#
#               Server Configuration               #
#==================================================#

HOST=localhost
PORT=3080

MONGO_URI=mongodb://127.0.0.1:27017/LibreChat

DOMAIN_CLIENT=http://localhost:3080
DOMAIN_SERVER=http://localhost:3080

NO_INDEX=true

#===============#
# JSON Logging  #
#===============#

# Use when process console logs in cloud deployment like GCP/AWS
CONSOLE_JSON=false

#===============#
# Debug Logging #
#===============#

DEBUG_LOGGING=true
DEBUG_CONSOLE=false

#=============#
# Permissions #
#=============#

# UID=1000
# GID=1000

#===============#
# Configuration #
#===============#
# Use an absolute path, a relative path, or a URL

# CONFIG_PATH="/alternative/path/to/librechat.yaml"

#===================================================#
#                     Endpoints                     #
#===================================================#

ENDPOINTS=openAI,assistants,anthropic,azureOpenAI,azureAssistants,gptPlugins
#azureOpenAI,bingAI,google,gptPlugins,anthropic

PROXY=

#===================================#
# Known Endpoints - librechat.yaml  #
#===================================#
# https://docs.librechat.ai/install/configuration/ai_endpoints.html

# GROQ_API_KEY=
# SHUTTLEAI_KEY=
# OPENROUTER_KEY=
# MISTRAL_API_KEY=
# ANYSCALE_API_KEY=
# FIREWORKS_API_KEY=
# PERPLEXITY_API_KEY=
# TOGETHERAI_API_KEY=
AZUREOPENAI_SWEDEN_API_KEY=my_azure_openai_key

#============#
# Anthropic  #
#============#

ANTHROPIC_API_KEY=my_anthropic_key
ANTHROPIC_MODELS=claude-3-5-sonnet-20240620,claude-3-opus-20240229,claude-3-haiku-20240307,claude-2.1,claude-2,claude-1.2,claude-1,claude-1-100k,claude-instant-1,claude-instant-1-100k
# ANTHROPIC_REVERSE_PROXY=

#============#
# Azure      #
#============#


RAG_AZURE_OPENAI_ENDPOINT=https://my_instance_name_sweden.openai.azure.com/
#DEBUG_RAG_API=true
RAG_AZURE_OPENAI_API_KEY=my_azure_openai_key
#RAG_AZURE_OPENAI_API_VERSION=2024-05-01-preview
EMBEDDINGS_PROVIDER=azure
EMBEDDINGS_MODEL=text-embedding-3-large


# Note: these variables are DEPRECATED
# Use the `librechat.yaml` configuration for `azureOpenAI` instead
# You may also continue to use them if you opt out of using the `librechat.yaml` configuration

# AZURE_OPENAI_DEFAULT_MODEL=gpt-3.5-turbo # Deprecated
# AZURE_OPENAI_MODELS=gpt-3.5-turbo,gpt-4 # Deprecated
# AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE # Deprecated
# AZURE_API_KEY= # Deprecated
# AZURE_OPENAI_API_INSTANCE_NAME= # Deprecated
# AZURE_OPENAI_API_DEPLOYMENT_NAME= # Deprecated
# AZURE_OPENAI_API_VERSION= # Deprecated
# AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME= # Deprecated
# AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME= # Deprecated
# PLUGINS_USE_AZURE="true" # Deprecated

#============#
# BingAI     #
#============#

BINGAI_TOKEN=user_provided
# BINGAI_HOST=https://cn.bing.com

#============#
# Google     #
#============#

GOOGLE_KEY=user_provided
# GOOGLE_MODELS=gemini-pro,gemini-pro-vision,chat-bison,chat-bison-32k,codechat-bison,codechat-bison-32k,text-bison,text-bison-32k,text-unicorn,code-gecko,code-bison,code-bison-32k
# GOOGLE_REVERSE_PROXY=

#============#
# OpenAI     #
#============#

OPENAI_API_KEY=my_openai_key
# OPENAI_MODELS=gpt-3.5-turbo-0125,gpt-3.5-turbo-0301,gpt-3.5-turbo,gpt-4,gpt-4-0613,gpt-4-vision-preview,gpt-3.5-turbo-0613,gpt-3.5-turbo-16k-0613,gpt-4-0125-preview,gpt-4-turbo-preview,gpt-4-1106-preview,gpt-3.5-turbo-1106,gpt-3.5-turbo-instruct,gpt-3.5-turbo-instruct-0914,gpt-3.5-turbo-16k

DEBUG_OPENAI=false

# TITLE_CONVO=false
# OPENAI_TITLE_MODEL=gpt-3.5-turbo

# OPENAI_SUMMARIZE=true
# OPENAI_SUMMARY_MODEL=gpt-3.5-turbo

# OPENAI_FORCE_PROMPT=true

# OPENAI_REVERSE_PROXY=

# OPENAI_ORGANIZATION=

#====================#
#   Assistants API   #
#====================#

ASSISTANTS_API_KEY=my_openai_assistant_key
ASSISTANTS_BASE_URL=
ASSISTANTS_MODELS=gpt-3.5-turbo-0125,gpt-3.5-turbo-16k-0613,gpt-3.5-turbo-16k,gpt-3.5-turbo,gpt-4,gpt-4-0314,gpt-4-32k-0314,gpt-4-0613,gpt-3.5-turbo-0613,gpt-3.5-turbo-1106,gpt-4-0125-preview,gpt-4-turbo-preview,gpt-4-1106-preview

#============#
# OpenRouter #
#============#

# OPENROUTER_API_KEY=

#============#
# Plugins    #
#============#

PLUGIN_MODELS=gpt-4,gpt-4-turbo-preview,gpt-4-0125-preview,gpt-4-1106-preview,gpt-4-0613,gpt-3.5-turbo,gpt-3.5-turbo-0125,gpt-3.5-turbo-1106,gpt-3.5-turbo-0613 
#I am wondering whether I would better leave the line above commented; especially because I want to force plugin to work 
# with azure openai, and I find it weird to have the openai models listed here... or maybe I should list only the deployment 
# names of my azure openai endpoint ?

DEBUG_PLUGINS=true

CREDS_KEY=593df_my_creds_key_04695
CREDS_IV=8984f_my_creds_iv_key34c1

# Azure AI Search
#-----------------
AZURE_AI_SEARCH_SERVICE_ENDPOINT=
AZURE_AI_SEARCH_INDEX_NAME=
AZURE_AI_SEARCH_API_KEY=

AZURE_AI_SEARCH_API_VERSION=
AZURE_AI_SEARCH_SEARCH_OPTION_QUERY_TYPE=
AZURE_AI_SEARCH_SEARCH_OPTION_TOP=
AZURE_AI_SEARCH_SEARCH_OPTION_SELECT=

# DALL·E
#----------------
DALLE_API_KEY=my_azure_openai_key
# DALLE3_API_KEY=
# DALLE2_API_KEY=
# DALLE3_SYSTEM_PROMPT=
# DALLE2_SYSTEM_PROMPT=
# DALLE_REVERSE_PROXY=
DALLE3_BASEURL=https://my_instance_name_sweden.openai.azure.com/
# DALLE2_BASEURL=

# DALL·E (via Azure OpenAI)
# Note: requires some of the variables above to be set
#----------------
DALLE3_AZURE_API_VERSION="2024-02-01"
# DALLE2_AZURE_API_VERSION=

# Google
#-----------------
GOOGLE_API_KEY=
GOOGLE_CSE_ID=

# SerpAPI
#-----------------
SERPAPI_API_KEY=

# Stable Diffusion
#-----------------
SD_WEBUI_URL=http://host.docker.internal:7860

# Tavily
#-----------------
TAVILY_API_KEY=

# Traversaal
#-----------------
TRAVERSAAL_API_KEY=

# WolframAlpha
#-----------------
WOLFRAM_APP_ID=

# Zapier
#-----------------
ZAPIER_NLA_API_KEY=

#==================================================#
#                      Search                      #
#==================================================#

SEARCH=true
MEILI_NO_ANALYTICS=true
MEILI_HOST=http://0.0.0.0:7700
MEILI_MASTER_KEY=my_master_key

#===================================================#
#                    User System                    #
#===================================================#

#========================#
# Moderation             #
#========================#

OPENAI_MODERATION=false
OPENAI_MODERATION_API_KEY=
# OPENAI_MODERATION_REVERSE_PROXY=

BAN_VIOLATIONS=true
BAN_DURATION=1000 * 60 * 60 * 2
BAN_INTERVAL=20

LOGIN_VIOLATION_SCORE=1
REGISTRATION_VIOLATION_SCORE=1
CONCURRENT_VIOLATION_SCORE=1
MESSAGE_VIOLATION_SCORE=1
NON_BROWSER_VIOLATION_SCORE=20

LOGIN_MAX=7
LOGIN_WINDOW=5
REGISTER_MAX=5
REGISTER_WINDOW=60

LIMIT_CONCURRENT_MESSAGES=true
CONCURRENT_MESSAGE_MAX=2

LIMIT_MESSAGE_IP=true
MESSAGE_IP_MAX=40
MESSAGE_IP_WINDOW=1

LIMIT_MESSAGE_USER=false
MESSAGE_USER_MAX=40
MESSAGE_USER_WINDOW=1

ILLEGAL_MODEL_REQ_SCORE=5

#========================#
# Balance                #
#========================#

CHECK_BALANCE=false

#========================#
# Registration and Login #
#========================#

ALLOW_EMAIL_LOGIN=true
ALLOW_REGISTRATION=true
ALLOW_SOCIAL_LOGIN=false
ALLOW_SOCIAL_REGISTRATION=false

SESSION_EXPIRY=1000 * 60 * 15
REFRESH_TOKEN_EXPIRY=(1000 * 60 * 60 * 24) * 7

JWT_SECRET=my_new_jwt_secret
JWT_REFRESH_SECRET=my_new_jwt_refresh_secret

# Discord
DISCORD_CLIENT_ID=
DISCORD_CLIENT_SECRET=
DISCORD_CALLBACK_URL=/oauth/discord/callback

# Facebook
FACEBOOK_CLIENT_ID=
FACEBOOK_CLIENT_SECRET=
FACEBOOK_CALLBACK_URL=/oauth/facebook/callback

# GitHub
GITHUB_CLIENT_ID=
GITHUB_CLIENT_SECRET=
GITHUB_CALLBACK_URL=/oauth/github/callback

# Google
GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=
GOOGLE_CALLBACK_URL=/oauth/google/callback

# OpenID
OPENID_CLIENT_ID=
OPENID_CLIENT_SECRET=
OPENID_ISSUER=
OPENID_SESSION_SECRET=
OPENID_SCOPE="openid profile email"
OPENID_CALLBACK_URL=/oauth/openid/callback
OPENID_REQUIRED_ROLE=
OPENID_REQUIRED_ROLE_TOKEN_KIND=
OPENID_REQUIRED_ROLE_PARAMETER_PATH=

OPENID_BUTTON_LABEL=
OPENID_IMAGE_URL=

#========================#
# Email Password Reset   #
#========================#

EMAIL_SERVICE=
EMAIL_HOST=
EMAIL_PORT=25
EMAIL_ENCRYPTION=
EMAIL_ENCRYPTION_HOSTNAME=
EMAIL_ALLOW_SELFSIGNED=
EMAIL_USERNAME=
EMAIL_PASSWORD=
EMAIL_FROM_NAME=
[email protected]

#========================#
# Firebase CDN           #
#========================#

FIREBASE_API_KEY=
FIREBASE_AUTH_DOMAIN=
FIREBASE_PROJECT_ID=
FIREBASE_STORAGE_BUCKET=
FIREBASE_MESSAGING_SENDER_ID=
FIREBASE_APP_ID=

#===================================================#
#                        UI                         #
#===================================================#

APP_TITLE="LibreChat"
CUSTOM_FOOTER="My custom footer"
HELP_AND_FAQ_URL=https://librechat.ai
# SHOW_BIRTHDAY_ICON=true

#==================================================#
#                      Others                      #
#==================================================#
#   You should leave the following commented out   #

# NODE_ENV=

# REDIS_URI=
# USE_REDIS=

# E2E_USER_EMAIL=
# E2E_USER_PASSWORD=

Thanks in advance for your help !

[fbaeumer]

Have you excluded or included plugins in your librechat.yml? In the sense of:

plugins

includedTools: ["calculator", "serpapi", "dalle"]

[djuillard]

No, as copy-pasted before, I have only set the following lines in my .env file :

ENDPOINTS=openAI,assistants,anthropic,azureOpenAI,azureAssistants,gptPlugins

...

PLUGIN_MODELS=gpt-4,gpt-4-turbo-preview,gpt-4-0125-preview,gpt-4-1106-preview,gpt-4-0613,gpt-3.5-turbo,gpt-3.5-turbo-0125,gpt-3.5-turbo-1106,gpt-3.5-turbo-0613 
#I am wondering whether I would better leave the line above commented; especially because I want to force plugin to work 
# with azure openai, and I find it weird to have the openai models listed here... or maybe I should list only the deployment 
# names of my azure openai endpoint ?

...

# DALL·E
#----------------
DALLE_API_KEY=my_azure_openai_key
# DALLE3_API_KEY=
# DALLE2_API_KEY=
# DALLE3_SYSTEM_PROMPT=
# DALLE2_SYSTEM_PROMPT=
# DALLE_REVERSE_PROXY=
DALLE3_BASEURL=https://my_instance_name_sweden.openai.azure.com/
# DALLE2_BASEURL=

And about my librechat.yaml file, I have onlyset the following :

endpoints:
  azureOpenAI:
    titleModel: "gpt-35-turbo"
    plugins: true
    assistants: true
    ....

Where am I supposed to add the array you mention ? right after that line ?

plugins: true

Thanks

[fbaeumer]

In your first post you wrote librechat.yml and copied in the .env. Hence my question.

In your .env, btw, the DALLE3_BASEURL looks wrong. Please double-check the docs:
https://www.librechat.ai/docs/configuration/azure#dall-e

In my case its:
DALLE3_BASEURL=https://XXXXX.openai.azure.com/openai/deployments/dall-e-3/ where "dall-e-3" is my deployment name.

However, since the store doesn't open for you at all, it could also be a JS error, does your browser spit something out when you press the button (Developer Console)?

For more information, see the Configuration Guide:
 https://docs.librechat.ai/install/configuration/custom_config.html

 Configuration version (required)
version: 1.0.6

 Cache settings: Set to true to enable caching
cache: true

 Plugins
includedTools: ["calculator", "serpapi", "dalle"]

Since "includedTools" is a root parameter, you can simply place it at the top level.

[djuillard]

Excellent, it works, thanks a lot.