How to configure RAG for Azure OpenAI?

[dareima]

Hi everyone,

today we've noticed that RAG only worked since we've added an OpenAI API Key to our .env file.

As our main endpoints are in Microsoft Azure OpenAI, we'd like to utilize our embeddings model there.
We gave the information for other providers provided in https://www.librechat.ai/docs/configuration/rag_api a go but weren't succesful.

Can someone describe which configurations we need to put in place to switch RAG to Azure OpenAI?

Any help would be much appreciated, thanks!

[dareima]

All good. Figured out how to do it, thanks to the documentation in the rag project: https://github.com/danny-avila/rag_api

Setting these params fixed it for us:

RAG_AZURE_OPENAI_API_KEY=nnn
RAG_AZURE_OPENAI_ENDPOINT=https://ourresrouce.openai.azure.com
EMBEDDINGS_PROVIDER=azure
EMBEDDINGS_MODEL=text-embedding-ada-002

In love with LibreChat @danny-avila 💯

[jmariquit-tvlk]

I tried this but am getting this error (note am not using OPENAI but using Azure OpenAI instead):

Value error, Must provide either the api_version argument or the OPENAI_API_VERSION environment variable [type=value_error, input_value={'azure_deployment': 'tex...one, 'model_kwargs': {}}, input_type=dict]

[dareima]

Hard to tell the cause without knowing more about the rest of your Azure OpenAI setup in LibreChat.

According to this error LibreChat is missing an API version setting. As API versions for the models are no longer set in .env but should be set in librechat.yaml, it will be good to see how your model setup there looks like.

Also, are you using modelSpecs or are you going without them?

[jmariquit-tvlk]

thanks for the input. this is what I have in my librechat.yaml (masked the values of the names and actual keys):

endpoints:
  azureOpenAI:
    titleModel: "current_model"
    plugins: true
    assistants: true
    titleConvo: true
    groups:
      - group: "xxx-yyy"
        apiKey: "xxx...yyy"
        instanceName: "xxx-yyy"
        version: "2024-05-01-preview"
        assistants: true
        models:
          gpt4o-2024-05-01:
            deploymentName: "chatgpt"

as for modelSpecs, I don't have anything defined. the endpoint definition does work though (i.e. I can chat with the model no issues except if I attach a file for the rag)

taking a look at the link I again, I will try to specify the RAG_AZURE_OPENAI_API_VERSION= var. will also play around with the modelSpecs.

[jmariquit-tvlk]

it worked! I was missing the RAG_AZURE_OPENAI_API_VERSION

I made no changes to my librechat.yaml (as above) and I did not need to look at modelSpecs either.
after i added the RAG_AZURE_OPENAI_API_VERSION to my .env it was able to connect properly.

here's my .env entry for those trying to get the RAG to work with Azure

# text-embedding-ada-002 and text-embedding-3-small also works (as long as you have them deployed as well)
EMBEDDINGS_PROVIDER=azure
EMBEDDINGS_MODEL=text-embedding-3-large

# replace with your key and endpoint of course
RAG_AZURE_OPENAI_API_KEY=xxx....yyy
RAG_AZURE_OPENAI_ENDPOINT=https://???.openai.azure.com

# endpoint had 2023-05-15 as a default but mine worked as well
RAG_AZURE_OPENAI_API_VERSION=2024-05-01-preview

thanks again.

[dareima]

it worked! I was missing the RAG_AZURE_OPENAI_API_VERSION

I made no changes to my librechat.yaml (as above) and I did not need to look at modelSpecs either. after i added the RAG_AZURE_OPENAI_API_VERSION to my .env it was able to connect properly.

here's my .env entry for those trying to get the RAG to work with Azure

# text-embedding-ada-002 and text-embedding-3-small also works (as long as you have them deployed as well)
EMBEDDINGS_PROVIDER=azure
EMBEDDINGS_MODEL=text-embedding-3-large

# replace with your key and endpoint of course
RAG_AZURE_OPENAI_API_KEY=xxx....yyy
RAG_AZURE_OPENAI_ENDPOINT=https://???.openai.azure.com

# endpoint had 2023-05-15 as a default but mine worked as well
RAG_AZURE_OPENAI_API_VERSION=2024-05-01-preview

thanks again.

Very nice, thanks for the update.
Interesting that we don't have to use RAG_AZURE_OPENAI_API_VERSION=2024-05-01-preview in our .env.

The only difference I can spot at a first glance is that we're also defining each model API version directly in librechat.yaml.

Like this:

endpoints:
  azureOpenAI:
    titleModel: "current_model"
    plugins: true
    assistants: true
    groups:
      - group: "swe-central" # arbitrary name
        apiKey: "${AZURE_OPEN_AI_API_KEY}"
        assistants: true
        instanceName: "aihubswe1234567890"
        version: "2024-05-01-preview"
        models:
          swe-gpt-4o:
            deploymentName: swe-gpt-4o
            version: "2024-02-15-preview"
          swe-gpt-4o-mini:
            deploymentName: swe-gpt-4o-mini
            version: "2024-02-15-preview"

I wouldn't have thought this might make a difference in regards to RAG.
Anyway, good to see it's working for you.

[rubenaster]

@dareima @jmariquit-tvlk does it still work for you?

With 0.7.8 when I want to upload a pdf to the current chat, I get the following error, even so all five env settings have been configured correctly:
Error processing file: RAG_API_URL not defined

I'm running on an App Service and setting RAG_API_URL to http://localhost:80 or http://host.docker.internal:8000/ gives the following error:

RAG API is either not running or not reachable at http://localhost:80, you may experience errors with file uploads.

[..]

Error uploading vectors An error occurred while setting up the request: File embedding failed.
Error processing file: File embedding failed.

[dareima]

@rubenaster can you show all RAG specific variables you have in place in your .env?
RAG still works fine for me in 0.7.8 but I do remember I had issues somewhere in between releases.

[rubenaster]

@dareima Sure, thanks for helping!

"name": "EMBEDDINGS_MODEL",
"value": "text-embedding-3-large"

"name": "EMBEDDINGS_PROVIDER",
"value": "azure"

"name": "PORT",
"value": "80"

"name": "RAG_API_URL",
"value": "http://localhost:80"

"name": "RAG_AZURE_OPENAI_API_KEY",
"value": "XXX"

"name": "RAG_AZURE_OPENAI_API_VERSION",
"value": "2025-02-01-preview"

"name": "RAG_AZURE_OPENAI_ENDPOINT",
"value": "https://myopenaiservice.openai.azure.com/"

And as I said, I'm running on an Azure App Service with container runtime.

[dareima]

@rubenaster in comparison, these are in the only environment variables we are using for RAG in LibreChat.
Meaning, any other 'RAG_' variable that exists, is either not used or commented out in our environment.

I remember we also ran into issues just by upgrading to newer releases and got similar errors like you've shown.
We then started to narrow it down and let LibreChat use its defaults, which made everything work again.

Hope that helps and also applies to Libre running as an App Service.
Ours is running in a dedicated Azure VM.

RAG_AZURE_OPENAI_ENDPOINT=https://ourendpoint.openai.azure.com
RAG_AZURE_OPENAI_API_VERSION=2025-03-01-preview
EMBEDDINGS_PROVIDER=azure
EMBEDDINGS_MODEL=text-embedding-3-large

[rubenaster]

@danny-avila even the setup provided by @dareima doesn't work on my end. I still get the errors mentioned above.

Could you please provide an example configuration like you'd expect it for an Azure App Service running container runtime and an Azure OpenAI service instance?

2025-05-22T04:58:31.001731943Z  2025-05-22 04:58:31 error: Error uploading vectors The server responded with status 404: Request failed with status code 404
2025-05-22T04:58:31.002152172Z  2025-05-22 04:58:31 error: [/files] Error processing file: Request failed with status code 404

[rubenaster]

@dareima do you need to have an additional container running that serves as the RAG API or is the LibreChat container enough? I don't understand why I get the mentioned issues.

[dareima]

@rubenaster absolutely, the default LibreChat Docker implementation consists of five containers.
Note that I am on the librechat-dev package, which is why dev is in the name of the LibreChat and the rag_api image.
For production we're using the regular librechat package as that always contains the latest stable release.
See packages here: https://github.com/danny-avila?tab=packages&repo_name=LibreChat

My images / containers in use:

ghcr.io/danny-avila/librechat-dev:latest LibreChat
ghcr.io/danny-avila/librechat-rag-api-dev-lite:latest rag_api
ankane/pgvector:latest vectordb
mongo chat-mongodb
getmeili/meilisearch:v1.12.3 chat-meilisearch

[JairajSustained]

Are you guys using the key and URL given in the overview section of "Azure AI Foundry" because I tried that and it didn't worked.
I mean where do I go in the below image to get those endpoints and the API KEY
I went to the overview.

[JairajSustained]

Found the solution
I had to deploy this model, then it started working
text-embedding-3-small

I'm wondering what I have to deploy this model to get it working but not in big AGI local deployment.

[dareima]

@JairajSustained to provide a simple answer on why you also need an embeddings model when performing RAG:

When using Retrieval-Augmented Generation (RAG) in AI systems, embeddings and chat completion models work together like a librarian and an expert storyteller. Here's why both are needed:

1. Embeddings act as the "librarian":

• They convert words into numerical codes that capture meaning (e.g., "dog" and "puppy" get similar codes).
• When you ask a question, embeddings quickly find relevant information by matching these meaning-based codes.
• Without embeddings, AI would only search for exact keywords ("Find every document with the word 'dog'") instead of understanding context.

2. Chat completions act as the "storyteller":

• Once embeddings find relevant information, the completion model (like GPT-4o) explains it in natural language.
• While completions are good at writing, they don't inherently know which facts to use – they need the librarian's help to avoid guessing.

Real-world analogy:
Imagine asking "Why do dogs bark?"

• Embeddings: Find entries about canine communication, dog behavior studies, and vet guides.
• Completions: Combine this information into a clear answer: "Dogs bark to communicate emotions like excitement or alert others to danger...".

This combination helps AI systems provide accurate, up-to-date answers instead of relying solely on what they memorized during training.