Need help to make tools work on Azure Assistants

[djuillard]

I have setup tools and plugin on my LibreChat instances. I want them to work for both AzureOpenAI and AzureAssistants endpoints, but so far it only works for AzureOpenAI.

Here is my conf of my .env file :

#=====================================================================#
#                       LibreChat Configuration                       #
#=====================================================================#
# Please refer to the reference documentation for assistance          #
# with configuring your LibreChat environment. The guide is           #
# available both online and within your local LibreChat               #
# directory:                                                          #
# Online: https://docs.librechat.ai/install/configuration/dotenv.html #
# Locally: ./docs/install/configuration/dotenv.md                     #
#=====================================================================#

#==================================================#
#               Server Configuration               #
#==================================================#

HOST=localhost
PORT=3080

MONGO_URI=mongodb://127.0.0.1:27017/LibreChat

DOMAIN_CLIENT=http://localhost:3080
DOMAIN_SERVER=http://localhost:3080

NO_INDEX=true

#===============#
# JSON Logging  #
#===============#

# Use when process console logs in cloud deployment like GCP/AWS
CONSOLE_JSON=false

#===============#
# Debug Logging #
#===============#

DEBUG_LOGGING=true
DEBUG_CONSOLE=false

#=============#
# Permissions #
#=============#

# UID=1000
# GID=1000

#===============#
# Configuration #
#===============#
# Use an absolute path, a relative path, or a URL

# CONFIG_PATH="/alternative/path/to/librechat.yaml"

#===================================================#
#                     Endpoints                     #
#===================================================#

ENDPOINTS=azureOpenAI,azureAssistants,gptPlugins
#azureOpenAI,bingAI,google,gptPlugins,anthropic

PROXY=

#===================================#
# Known Endpoints - librechat.yaml  #
#===================================#
# https://docs.librechat.ai/install/configuration/ai_endpoints.html

AZUREOPENAI_SWEDEN_API_KEY=my_azure_openai_key


#============#
# Azure      #
#============#


RAG_AZURE_OPENAI_ENDPOINT=https://myinstance.openai.azure.com/
#DEBUG_RAG_API=true
RAG_AZURE_OPENAI_API_KEY=my_azure_openai_key
#RAG_AZURE_OPENAI_API_VERSION=2024-05-01-preview
EMBEDDINGS_PROVIDER=azure
EMBEDDINGS_MODEL=text-embedding-3-large




DEBUG_PLUGINS=true



# DALL·E
#----------------
# DALLE_API_KEY=
DALLE3_API_KEY=my_azure_openai_key
# DALLE2_API_KEY=
# DALLE3_SYSTEM_PROMPT=
# DALLE2_SYSTEM_PROMPT=
# DALLE_REVERSE_PROXY=
DALLE3_BASEURL=https://myinstance.openai.azure.com/openai/deployments/Dall-E3/
# DALLE2_BASEURL=

# DALL·E (via Azure OpenAI)
# Note: requires some of the variables above to be set
#----------------
DALLE3_AZURE_API_VERSION="2024-02-01"
# DALLE2_AZURE_API_VERSION=

# Google
#-----------------
GOOGLE_SEARCH_API_KEY=my_key
GOOGLE_CSE_ID=my_id

# SerpAPI
#-----------------
SERPAPI_API_KEY=

# Stable Diffusion
#-----------------
SD_WEBUI_URL=http://host.docker.internal:7860

# Tavily
#-----------------
TAVILY_API_KEY=my_key

# Traversaal
#-----------------
TRAVERSAAL_API_KEY=my_key

# WolframAlpha
#-----------------
WOLFRAM_APP_ID=

# Zapier
#-----------------
ZAPIER_NLA_API_KEY=

Here is my conf of my librechat.yaml file :

# For more information, see the Configuration Guide:
# https://www.librechat.ai/docs/configuration/librechat_yaml

# Configuration version (required)
version: 1.1.4

# Cache settings: Set to true to enable caching
cache: true

includedTools: ["calculator", "google", "dalle", "tavily_search_results_json", "traversaal_search"]

# Custom interface configuration
interface:
  # Privacy policy settings
  privacyPolicy:
    externalUrl: 'https://librechat.ai/privacy-policy'
    openNewTab: true

  # Terms of service
  termsOfService:
    externalUrl: 'https://librechat.ai/tos'
    openNewTab: true
  endpointsMenu: true
  modelSelect: true
  parameters: true
  sidePanel: true
  presets: true


# Example Registration Object Structure (optional)
registration:
  socialLogins: ['github', 'google', 'discord', 'openid', 'facebook']

endpoints:
  azureAssistants:
     disableBuilder: false # Disable Assistants Builder Interface by setting to `true`
     pollIntervalMs: 750  # Polling interval for checking assistant updates
     timeoutMs: 180000  # Timeout for assistant operations
  #   # Should only be one or the other, either `supportedIds` or `excludedIds`
  #   supportedIds: ["asst_supportedAssistantId1", "asst_supportedAssistantId2"]
  #   # excludedIds: ["asst_excludedAssistantId"]
  #   # (optional) Models that support retrieval, will default to latest known OpenAI models that support the feature
     retrievalModels: ["gpt-4o"]
  #   # (optional) Assistant Capabilities available to all users. Omit the ones you wish to exclude. Defaults to list below.
     capabilities: ["code_interpreter", "actions", "tools", "image_vision"]
     privateAssistants: true

  azureOpenAI:
    titleModel: "gpt-35-turbo"
    plugins: true
    assistants: true
    groups:
      - group: "Azure_OpenAI_RG1"
        apiKey: "${AZUREOPENAI_SWEDEN_API_KEY}"
        instanceName: "myinstance"
        deploymentName: "GPT-4-SWEDEN"
        assistants: true
        version: "2024-05-01-preview" # Update with the correct version if needed
        models:
          gpt-4:
            deploymentName: "GPT-4-SWEDEN"
            version: "2024-05-01-preview"
          gpt-35-turbo: 
            deploymentName: "GPT-35-turbo"
            version: "2024-05-01-preview"
          gpt-4o: 
            deploymentName: "GPT-4o"
            version: "2024-05-01-preview"
          whisper:
            deploymentName: "whisper"
            version: "2024-05-01-preview"          


fileConfig:
  endpoints:
    azureOpenAI:
      fileLimit: 3
      fileSizeLimit: 10
      totalSizeLimit: 15

I can see in the dropdown list the plugin :

And it works fine.

But when I select the azure assistant endpoint, and try to add tool, that's strange, I only see "Calculator" and nothing else....

I guess there is something wrong in my config files, but I cannot figure out what !

[bnord01]

The filtering for assistant tools in ToolService, is based on the plugin filename in api/app/clients/tools/structured. I didn't try it but looking at the code you can try including both the plugin id as well as the filename (e.g. TraversaalSearch.js) to the included tools.

[mansiibm]

Hi did the toolname/filename worked i tried the following - includedTools: ["calculator","stable-diffusion/StableDiffusion.js","google/GoogleSearch.js","wolfram/Wolfram.js","dalle/DALLE3.js","azure-ai-search/AzureAISearch.js","tavily_search_results_json/TavilySearchResults.js","traversaal_search/TraversaalSearch.js","dall-e/DALL-E.js","image_prompt_enhancer","web-browser"]
but it is not working please let me know if I am wrong

[bnord01]

You may try the latest development version from main and include only the pluginKey as listed in the manifest.json.

Or (with older versions) you include both the pluginKey as well as the (unqualified) file name as separate entities in the list.

[mansiibm]

Hi , Thankyou for replying you mean something like this- includedTools: ["calculator","stable-diffusion,"google","wolfram,"dalle","azure-ai-search","tavily_search_results_json","traversaal_search","dall-e","image_prompt_enhancer","web-browser"] And this should work? I tried this method above but it did not work. Regards, Mansi Gusain.

…

________________________________ From: Benedikt Nordhoff ***@***.***> Sent: Wednesday, July 17, 2024 4:27 PM To: danny-avila/LibreChat ***@***.***> Cc: Mansi Gusain ***@***.***>; Comment ***@***.***> Subject: [EXTERNAL] Re: [danny-avila/LibreChat] Need help to make tools work on Azure Assistants (Discussion #3255) You may try the latest development version from main and include only the pluginKey as listed in the manifest. json. Or (with older versions) you include both the pluginKey as well as the (unqualified) file name as separate entities in the list.  You may try the latest development version from main and include only the pluginKey as listed in the manifest.json<https://github.com/danny-avila/LibreChat/blob/main/api/app/clients/tools/manifest.json>. Or (with older versions) you include both the pluginKey as well as the (unqualified) file name as separate entities in the list. — Reply to this email directly, view it on GitHub<#3255 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHRPDVV6Y3ZOATCRH5IN2NLZMZEZRAVCNFSM6AAAAABKKFMY4WVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBXGIZTANY>. You are receiving this because you commented.Message ID: ***@***.***> Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

[bnord01]

No, that should only work using the dev version https://github.com/danny-avila/LibreChat/pkgs/container/librechat-dev

[mansiibm]

Hi, Also, I am using this image - https://ghcr.io/danny-avila/librechat-api:v0.7.4-rc1. Regards, Mansi Gusain.

…

________________________________ From: Mansi Gusain ***@***.***> Sent: Wednesday, July 17, 2024 4:31 PM To: danny-avila/LibreChat ***@***.***> Subject: Re: [EXTERNAL] Re: [danny-avila/LibreChat] Need help to make tools work on Azure Assistants (Discussion #3255) Hi , Thankyou for replying you mean something like this- includedTools: ["calculator","stable-diffusion,"google","wolfram,"dalle","azure-ai-search","tavily_search_results_json","traversaal_search","dall-e","image_prompt_enhancer","web-browser"] And this should work? I tried this method above but it did not work. Regards, Mansi Gusain.

________________________________ From: Benedikt Nordhoff ***@***.***> Sent: Wednesday, July 17, 2024 4:27 PM To: danny-avila/LibreChat ***@***.***> Cc: Mansi Gusain ***@***.***>; Comment ***@***.***> Subject: [EXTERNAL] Re: [danny-avila/LibreChat] Need help to make tools work on Azure Assistants (Discussion #3255) You may try the latest development version from main and include only the pluginKey as listed in the manifest. json. Or (with older versions) you include both the pluginKey as well as the (unqualified) file name as separate entities in the list.  You may try the latest development version from main and include only the pluginKey as listed in the manifest.json<https://github.com/danny-avila/LibreChat/blob/main/api/app/clients/tools/manifest.json>. Or (with older versions) you include both the pluginKey as well as the (unqualified) file name as separate entities in the list. — Reply to this email directly, view it on GitHub<#3255 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHRPDVV6Y3ZOATCRH5IN2NLZMZEZRAVCNFSM6AAAAABKKFMY4WVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBXGIZTANY>. You are receiving this because you commented.Message ID: ***@***.***> Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

[mansiibm]

Okay so not possible with the stable tag?? Always has to be dev image or if new version is released then this will be sorted ?? Please let me know . Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Benedikt Nordhoff ***@***.***> Sent: Wednesday, July 17, 2024 6:59:05 PM To: danny-avila/LibreChat ***@***.***> Cc: Mansi Gusain ***@***.***>; Comment ***@***.***> Subject: [EXTERNAL] Re: [danny-avila/LibreChat] Need help to make tools work on Azure Assistants (Discussion #3255) No, that should only work using the dev version https: //github. com/danny-avila/LibreChat/pkgs/container/librechat-dev — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented. Message No, that should only work using the dev version https://github.com/danny-avila/LibreChat/pkgs/container/librechat-dev<https://github.com/danny-avila/LibreChat/pkgs/container/librechat-dev> — Reply to this email directly, view it on GitHub<#3255 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHRPDVS3ETABJ7KESTFLEUTZMZWSDAVCNFSM6AAAAABKKFMY4WVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBXGM3TQMQ>. You are receiving this because you commented.Message ID: ***@***.***> Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

[bnord01]

As mentioned with the older versions you can try including both the pluginKey as well as the (unqualified) file name as separate entities in the list. E.g. includedTools: ["google","GoogleSearch.js", ... ]

[mansiibm]

Okay thanks I will try this Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Benedikt Nordhoff ***@***.***> Sent: Wednesday, July 17, 2024 7:09:59 PM To: danny-avila/LibreChat ***@***.***> Cc: Mansi Gusain ***@***.***>; Comment ***@***.***> Subject: [EXTERNAL] Re: [danny-avila/LibreChat] Need help to make tools work on Azure Assistants (Discussion #3255) As mentioned with the older versions you can try including both the pluginKey as well as the (unqualified) file name as separate entities in the list. E. g. includedTools: ["google","GoogleSearch. js", .. . ] — Reply to this email directly, As mentioned with the older versions you can try including both the pluginKey as well as the (unqualified) file name as separate entities in the list. E.g. includedTools: ["google","GoogleSearch.js", ... ] — Reply to this email directly, view it on GitHub<#3255 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHRPDVTPXOMVXGNTC54D4YTZMZX27AVCNFSM6AAAAABKKFMY4WVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBXGM4TAMI>. You are receiving this because you commented.Message ID: ***@***.***> Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

[mansiibm]

Thankyou so much for your help , it did work - the config is as- includedTools: ["traversaal_search","TraversaalSearch.js","google","GoogleSearch.js","wolfram","Wolfram.js","dalle","DALLE3.js","tavily_search_results_json","TavilySearchResults.js","calculator","stable-diffusion","StableDiffusion.js","azure-ai-search","AzureAISearch.js","image_prompt_enhancer","image_prompt_enhancer.json","web_search","web_search.json","dall-e","DALL-E.js","web-browser"]