Enable OCR in librechat

[jackdorney1999]

Hi there,

I am trying to utilise the OCR capabilities by making an addition to the librechat.yaml file:

# For more information, see the Configuration Guide:
# https://www.librechat.ai/docs/configuration/librechat_yaml

# Configuration version (required)
version: 1.2.1

# Cache settings: Set to true to enable caching
cache: false

# File strategy s3/firebase
# fileStrategy: "s3"

# Custom interface configuration
interface:
  customWelcome: "Welcome to LibreChat! Enjoy your experience."
  # Privacy policy settings
  privacyPolicy:
    externalUrl: 'https://librechat.ai/privacy-policy'
    openNewTab: true

  # Terms of service
  termsOfService:
    externalUrl: 'https://librechat.ai/tos'
    openNewTab: true
    modalAcceptance: true
    modalTitle: "Terms of Service for LibreChat"
    modalContent: |
      # Terms and Conditions for LibreChat

      *Effective Date: February 18, 2024*

      Welcome to LibreChat, the informational website for the open-source AI chat platform, available at https://librechat.ai. These Terms of Service ("Terms") govern your use of our website and the services we offer. By accessing or using the Website, you agree to be bound by these Terms and our Privacy Policy, accessible at https://librechat.ai//privacy.

      ## 1. Ownership

      Upon purchasing a package from LibreChat, you are granted the right to download and use the code for accessing an admin panel for LibreChat. While you own the downloaded code, you are expressly prohibited from reselling, redistributing, or otherwise transferring the code to third parties without explicit permission from LibreChat.

      ## 2. User Data

      We collect personal data, such as your name, email address, and payment information, as described in our Privacy Policy. This information is collected to provide and improve our services, process transactions, and communicate with you.

      ## 3. Non-Personal Data Collection

      The Website uses cookies to enhance user experience, analyze site usage, and facilitate certain functionalities. By using the Website, you consent to the use of cookies in accordance with our Privacy Policy.

      ## 4. Use of the Website

      You agree to use the Website only for lawful purposes and in a manner that does not infringe the rights of, restrict, or inhibit anyone else's use and enjoyment of the Website. Prohibited behavior includes harassing or causing distress or inconvenience to any person, transmitting obscene or offensive content, or disrupting the normal flow of dialogue within the Website.

      ## 5. Governing Law

      These Terms shall be governed by and construed in accordance with the laws of the United States, without giving effect to any principles of conflicts of law.

      ## 6. Changes to the Terms

      We reserve the right to modify these Terms at any time. We will notify users of any changes by email. Your continued use of the Website after such changes have been notified will constitute your consent to such changes.

      ## 7. Contact Information

      If you have any questions about these Terms, please contact us at [email protected].

      By using the Website, you acknowledge that you have read these Terms of Service and agree to be bound by them.

  endpointsMenu: true
  modelSelect: true
  parameters: true
  sidePanel: true
  presets: true
  prompts: true
  bookmarks: true
  multiConvo: true
  agents: true

# Example Registration Object Structure (optional)
registration:
  socialLogins: ['github', 'google', 'discord', 'openid', 'facebook', 'apple']
  # allowedDomains:
  # - "gmail.com"


# Example Balance settings
# balance:
#   enabled: false
#   startBalance: 20000
#   autoRefillEnabled: false
#   refillIntervalValue: 30
#   refillIntervalUnit: 'days'
#   refillAmount: 10000

# speech:
#   tts:
#     openai:
#       url: ''
#       apiKey: '${TTS_API_KEY}'
#       model: ''
#       voices: ['']

#
#   stt:
#     openai:
#       url: ''
#       apiKey: '${STT_API_KEY}'
#       model: ''

# rateLimits:
#   fileUploads:
#     ipMax: 100
#     ipWindowInMinutes: 60  # Rate limit window for file uploads per IP
#     userMax: 50
#     userWindowInMinutes: 60  # Rate limit window for file uploads per user
#   conversationsImport:
#     ipMax: 100
#     ipWindowInMinutes: 60  # Rate limit window for conversation imports per IP
#     userMax: 50
#     userWindowInMinutes: 60  # Rate limit window for conversation imports per user

# Example Actions Object Structure
actions:
  allowedDomains:
    - "swapi.dev"
    - "librechat.ai"
    - "google.com"

mcpServers:
  Tools:
    type: sse
    url: http://host.docker.internal:8000/sse



# Example MCP Servers Object Structure
# mcpServers:
#   everything:
#     # type: sse # type can optionally be omitted
#     url: http://localhost:3001/sse
#     timeout: 60000  # 1 minute timeout for this server, this is the default timeout for MCP servers.
#   puppeteer:
#     type: stdio
#     command: npx
#     args:
#       - -y
#       - "@modelcontextprotocol/server-puppeteer"
#     timeout: 300000  # 5 minutes timeout for this server
#   filesystem:
#     # type: stdio
#     command: npx
#     args:
#       - -y
#       - "@modelcontextprotocol/server-filesystem"
#       - /home/user/LibreChat/
#     iconPath: /home/user/LibreChat/client/public/assets/logo.svg
#   mcp-obsidian:
#     command: npx
#     args:
#       - -y
#       - "mcp-obsidian"
#       - /path/to/obsidian/vault

# Definition of custom endpoints
endpoints:
  agents:
    disableBuilder: false
    capabilities: ["execute_code", "file_search", "actions", "tools", "ocr"]
  azureAssistants:
    pollIntervalMs: 500
    timeoutMs: 10000
    capabilities: ["code_interpreter", "actions", "tools", "image_vision"]
  azureOpenAI:
    titleModel: "gpt-4o"
    groups:
      - group: "azure-default"
        apiKey: "${WEU_AZURE_API_KEY}"
        instanceName: "dev-weu-azure-openai"  # without `.openai.azure.com`
        version: "2024-05-01-preview"
        models:
          gpt-4o:
            deploymentName: gpt-4o
          gpt-3.5-turbo:
            deploymentName: gpt-35-turbo
  custom:
    - name: 'databricks-llm'
      apiKey: '${DATABRICKS_API_KEY}'
      baseURL: <databricks_url>
      # headers:
      #   Authorization: "Bearer ${DATABRICKS_API_KEY}"
      #   Content-Type: "application/json"
      models:
        default: ["databricks-meta-llama-3-3-70b-instruct"]
        fetch: false
      titleConvo: true
      titleModel: "databricks-meta-llama-3-3-70b-instruct"
      modelDisplayLabel: 'Databricks Llama 3.3'
      iconURL: 'https://companieslogo.com/img/orig/databricks.D-0e162e58.png?t=1720244494'

ocr:
   mistralModel: "mistral-ocr-latest"
   apiKey: "${OCR_API_KEY}"
   baseURL: "${OCR_BASEURL}"
   strategy: "mistral_ocr"

    # Groq Example
    # - name: 'groq'
    #   apiKey: '${GROQ_API_KEY}'
    #   baseURL: 'https://api.groq.com/openai/v1/'
    #   models:
    #     default:
    #       [
    #         'llama3-70b-8192',
    #         'llama3-8b-8192',
    #         'llama2-70b-4096',
    #         'mixtral-8x7b-32768',
    #         'gemma-7b-it',
    #       ]
    #     fetch: false
    #   titleConvo: true
    #   titleModel: 'mixtral-8x7b-32768'
    #   modelDisplayLabel: 'groq'

    # Mistral AI Example
    # - name: 'Mistral' # Unique name for the endpoint
      # For `apiKey` and `baseURL`, you can use environment variables that you define.
      # recommended environment variables:
      # apiKey: '${MISTRAL_API_KEY}'
      # baseURL: 'https://api.mistral.ai/v1'

      # Models configuration
      # models:
      #   # List of default models to use. At least one value is required.
      #   default: ['mistral-tiny', 'mistral-small', 'mistral-medium']
      #   # Fetch option: Set to true to fetch models from API.
      #   fetch: true # Defaults to false.

      # Optional configurations

      # Title Conversation setting
#       titleConvo: true # Set to true to enable title conversation

#       # Title Method: Choose between "completion" or "functions".
#       # titleMethod: "completion"  # Defaults to "completion" if omitted.

#       # Title Model: Specify the model to use for titles.
#       titleModel: 'mistral-tiny' # Defaults to "gpt-3.5-turbo" if omitted.

#       # Summarize setting: Set to true to enable summarization.
#       # summarize: false

#       # Summary Model: Specify the model to use if summarization is enabled.
#       # summaryModel: "mistral-tiny"  # Defaults to "gpt-3.5-turbo" if omitted.

#       # Force Prompt setting: If true, sends a `prompt` parameter instead of `messages`.
#       # forcePrompt: false

#       # The label displayed for the AI model in messages.
#       modelDisplayLabel: 'Mistral' # Default is "AI" when not set.

#       # Add additional parameters to the request. Default params will be overwritten.
#       # addParams:
#       # safe_prompt: true # This field is specific to Mistral AI: https://docs.mistral.ai/api/

#       # Drop Default params parameters from the request. See default params in guide linked below.
#       # NOTE: For Mistral, it is necessary to drop the following parameters or you will encounter a 422 Error:
#       dropParams: ['stop', 'user', 'frequency_penalty', 'presence_penalty']

#     # OpenRouter Example
#     - name: 'OpenRouter'
#       # For `apiKey` and `baseURL`, you can use environment variables that you define.
#       # recommended environment variables:
#       apiKey: '${OPENROUTER_KEY}'
#       baseURL: 'https://openrouter.ai/api/v1'
#       models:
#         default: ['meta-llama/llama-3-70b-instruct']
#         fetch: true
#       titleConvo: true
#       titleModel: 'meta-llama/llama-3-70b-instruct'
#       # Recommended: Drop the stop parameter from the request as Openrouter models use a variety of stop tokens.
#       dropParams: ['stop']
#       modelDisplayLabel: 'OpenRouter'

#     # Portkey AI Example
#     - name: "Portkey"
#       apiKey: "dummy"
#       baseURL: 'https://api.portkey.ai/v1'
#       headers:
#           x-portkey-api-key: '${PORTKEY_API_KEY}'
#           x-portkey-virtual-key: '${PORTKEY_OPENAI_VIRTUAL_KEY}'
#       models:
#           default: ['gpt-4o-mini', 'gpt-4o', 'chatgpt-4o-latest']
#           fetch: true
#       titleConvo: true
#       titleModel: 'current_model'
#       summarize: false
#       summaryModel: 'current_model'
#       forcePrompt: false
#       modelDisplayLabel: 'Portkey'
#       iconURL: https://images.crunchbase.com/image/upload/c_pad,f_auto,q_auto:eco,dpr_1/rjqy7ghvjoiu4cd1xjbf
# # fileConfig:
#   endpoints:
#     assistants:
#       fileLimit: 5
#       fileSizeLimit: 10  # Maximum size for an individual file in MB
#       totalSizeLimit: 50  # Maximum total size for all files in a single request in MB
#       supportedMimeTypes:
#         - "image/.*"
#         - "application/pdf"
#     openAI:
#       disabled: true  # Disables file uploading to the OpenAI endpoint
#     default:
#       totalSizeLimit: 20
#     YourCustomEndpointName:
#       fileLimit: 2
#       fileSizeLimit: 5
#   serverFileSizeLimit: 100  # Global server file size limit in MB
#   avatarSizeLimit: 2  # Limit for user avatar image size in MB
# # See the Custom Configuration Guide for more information on Assistants Config:
# # https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/assistants_endpoint

I have tried to add ocr as capability for the agents, but was met with this error:

LibreChat         | 2025-04-14 14:21:44 error: Invalid custom config file at /app/librechat.yaml:
LibreChat         | {
LibreChat         |   "issues": [
LibreChat         |     {
LibreChat         |       "received": "ocr",
LibreChat         |       "code": "invalid_enum_value",
LibreChat         |       "options": [
LibreChat         |         "hide_sequential_outputs",
LibreChat         |         "end_after_tools",
LibreChat         |         "execute_code",
LibreChat         |         "file_search",
LibreChat         |         "artifacts",
LibreChat         |         "actions",
LibreChat         |         "tools"
LibreChat         |       ],
LibreChat         |       "path": [
LibreChat         |         "endpoints",
LibreChat         |         "agents",
LibreChat         |         "capabilities",
LibreChat         |         4
LibreChat         |       ],
LibreChat         |       "message": "Invalid enum value. Expected 'hide_sequential_outputs' | 'end_after_tools' | 'execute_code' | 'file_search' | 'artifacts' | 'actions' | 'tools', received 'ocr'"
LibreChat         |     }
LibreChat         |   ],
LibreChat         |   "name": "ZodError"
LibreChat         | }

Additionally, I have set these in the .env file:

OCR_API_KEY=<mistral_key>
OCR_BASEURL=https://api.mistral.ai/v1

I was wondering how I can get around this? I notice that the version of librechat.yaml is 1.2.1, and ocr was supported from >1.2.2 however if I try and change this version, I get this when starting the docker container:

LibreChat         | Outdated Config version: 1.2.3
LibreChat         | Latest version: 1.2.1

Any guidance on this would be great!

Thanks

[danny-avila]

Here's how to update:

Updating Instructions (docker):

# Linux command to Remove all existing images
docker images -a | grep "librechat" | awk '{print $3}' | xargs docker rmi

# Windows Powershell
docker images -a | findstr "librechat" | ForEach-Object { docker rmi $_.Split()[2] }

Then follow the directions here: https://www.librechat.ai/docs/local/docker#update-librechat

[jackdorney1999]

Hi there,

Thanks for this, I realised that I was not pulling from the dev image hence I was not seeing the updated config options

Thanks!