How to use AzureChatOpenAI(o4-mini) with "llm_factory" for judge LLM?

[a5512167086]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
Hi,

I’m migrating my RAGAS evaluation pipeline from an older gpt-3.5-turbo setup to a new gpt-o4-mini deployment hosted on Azure OpenAI.

Previously, I used LangchainLLMWrapper with a langchain_openai.AzureChatOpenAI client as the LLM judge. After upgrading, I’m seeing both:

1. A deprecation warning suggesting I should use llm_factory instead of LangchainLLMWrapper:

DeprecationWarning: LangchainLLMWrapper is deprecated and will be removed in a future version. Use llm_factory instead: from openai import OpenAI; from ragas.llms import llm_factory; llm = llm_factory('gpt-4o-mini', client=OpenAI(api_key='...'))

2. A runtime error related to the temperature parameter when using my Azure deployment:

BadRequestError(Error code: 400 - {'error': {'message': "Unsupported value: 'temperature' does not support 0.01 with this model. Only the default (1) value is supported.", 'type': 'invalid_request_error', 'param': 'temperature', 'code': 'unsupported_value'}})

It looks like LangchainLLMWrapper (or the RAGAS internals using it) is setting a non-default temperature for the judge LLM, but on Azure the gpt-4o-mini deployment only supports the default temperature = 1. This leads to the unsupported_value error.

My questions are:

1. What is the recommended way to configure RAGAS to use an existing AzureChatOpenAI client as the LLM judge?
   The deprecation warning suggests using llm_factory, but the example only shows the standard OpenAI client and not an Azure OpenAI client.

2. If Azure OpenAI does not yet work with llm_factory, is there a recommended workaround to avoid the temperature issue (e.g. a way to force RAGAS not to override the temperature), or is support for Azure OpenAI planned in a future release?

Code Examples

from langchain_openai import AzureChatOpenAI
from ragas.llms import LangchainLLMWrapper
from app.configs.config import (
    AZURE_OPENAI_API_VERSION,
    AZURE_OPENAI_DEPLOYMENT,
    EMBEDDING_MODEL,
)

azure_llm = AzureChatOpenAI(
    azure_deployment='o4-mini',
    openai_api_version='2024-12-01-preview',
    temperature=1,
    model_kwargs={"max_completion_tokens": 4096},
)

wrapped_llm = LangchainLLMWrapper(azure_llm)

result = evaluate(
    dataset=ragas_dataset,
    metrics=metrics,
    llm=wrapped_llm ,
    embeddings=embeddings,
)

[anistark]

@a5512167086 Thanks for reporting the issue.

LangchainLLMWrapper is deprecated and will be removed in future. Try using llm_factory.
With Ragas 0.3.9 you can pass custom temperature.
Example for Azure OpenAI:

from openai import AzureOpenAI
from ragas.llms import llm_factory

client = AzureOpenAI(
    api_key="...",
    api_version="...",
    azure_endpoint="..."
)

# Pass temperature=1.0 to override the default 0.01
llm = llm_factory(
    "gpt-4o-mini",
    client=client,
    provider="openai",
    temperature=1.0
)

Will take into note to add custom support for Azure's gpt-4o-mini as well by next release.

[anistark]

Closing as implemented in #2424