AzureChatOpenAI not working with cognitive regional endpoints?

[sorin-costea]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_openai import AzureChatOpenAI
azure_llm = AzureChatOpenAI(
    azure_deployment='azure/my_deployment',
    azure_endpoint='https://my_region.api.cognitive.microsoft.com/',
    api_version='some_version',
    api_key='my_key'
)

Description

I tried also with the whole endpoint as Azure shows it: https://my_region.api.cognitive.microsoft.com/openai/deployments/my_deployment/chat/completions?api-version=some_version also with the shortened https://my_region.api.cognitive.microsoft.com/openai/ or even https://my_region.api.cognitive.microsoft.com/ yet the error is always the same:

llm.py-llm:161 - ERROR: LiteLLM call failed: litellm.APIError: AzureException APIError - argument of type 'NoneType' is not iterable Traceback (most recent call last):
  File ".\pypoetry\Cache\virtualenvs\cv-crew-B-au1vhR-py3.12\Lib\site-packages\litellm\llms\AzureOpenAI\azure.py", line 607, in completion
    if "gateway.ai.cloudflare.com" in api_base:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

As there's no mention of the new endpoint style in the documentation or in the chatbot, could it be it's not supported yet by some component in the stack? Or I just don't get it...

System Info

langchain==0.2.16
langchain-cohere==0.1.9
langchain-community==0.2.17
langchain-core==0.3.7
langchain-experimental==0.0.65
langchain-openai==0.2.1
langchain-text-splitters==0.2.4

python 3.12.5
windows 11

[dosubot[bot]]

Hey @sorin-costea, 👋 I'm here to help you with your technical issues and questions! I'm a bot and I can assist with bug fixes, answering questions, and getting you set up as a contributor.

I found some similar discussions and issues that might help you resolve the problem with AzureChatOpenAI and cognitive regional endpoints.

1. Problem with initializing AzureOpenAI() - This discussion highlights potential issues with the openai_api_base and deployment_name parameters. Ensure you are using azure_endpoint instead of openai_api_base and do not specify both deployment_name and openai_api_base at the same time ^[1].

2. Replication of a complex Azure OpenAI request in langchain - Switching to ChatOpenAI resolved the issue for another user ^[2].

3. openai.error.InvalidRequestError: The API deployment for this resource does not exist - This issue suggests ensuring the openai_api_base URL is correctly formatted and points to the correct Azure resource. Also, verify the deployment_name matches exactly with the one set up in Azure ^[3].

Here's a corrected example based on the information from these discussions:

from langchain_openai import AzureChatOpenAI

azure_llm = AzureChatOpenAI(
    azure_deployment='my_deployment',
    azure_endpoint='https://my_region.api.cognitive.microsoft.com/',
    api_version='some_version',
    api_key='my_key'
)

Ensure that:

• azure_deployment is the name of your deployment.
• azure_endpoint is correctly formatted without additional paths.
• api_version and api_key are correctly specified.

If the issue persists, consider checking if the endpoint style is supported by the current version of LangChain you are using. Upgrading to the latest version might also help.

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

Could you provide more details about the exact version of litellm you are using? Additionally, have you tried reaching out to the maintainers of litellm to confirm if the regional endpoint style is supported? It might also help to check if there are any specific configuration options or updates related to Azure endpoints in their documentation or release notes.

To continue talking to Dosu, mention @dosu.

[sorin-costea]

@dosu litellm==1.48.2
And yes I just reached to them to ask about the endpoint style. Waiting for an answer...

[dosubot[bot]]

Hey @sorin-costea! 😊 I'm here to help with technical questions and bug fixes, but it looks like you're already on the right track by reaching out to the maintainers. If you think I should have more info on this, feel free to contact the Dosu Team. Keep me posted on how it goes!

To continue talking to Dosu, mention @dosu.

[sorin-costea]

@dosu I checked with litellm and yes they do support the regional cognitive endpoint style: BerriAI/litellm#5995
I am initializing AzureChatOpenAI the indicated way, yet LiteLLM doesn't receive the needed api_base - so I must assume the AzureChatOpenAI is not handling the parameters correctly?

[dosubot[bot]]

It sounds like there might be an issue with how AzureChatOpenAI is passing parameters to litellm. Could you share the exact code snippet you're using to initialize AzureChatOpenAI? This might help identify if there's a misconfiguration or if something needs to be adjusted in the code. Additionally, have you checked if there are any updates or patches for AzureChatOpenAI that address this issue?

To continue talking to Dosu, mention @dosu.

[sorin-costea]

@dosu If I initialize the LLM like this, everything works fine. So LiteLLM is definitely fine, my endpoint also fine, the only issue is I cannot use AzureChatOpenAI for it.

from crewai import LLM
azure_llm = LLM(
    model="azure/my_deployment",
    base_url="https://my_region.api.cognitive.microsoft.com/openai/deployments/my_deployment/chat/completions?api-version=some_version",
    api_key=api_key
)

For reference, the problem code is:

from langchain_openai import AzureChatOpenAI
azure_llm = AzureChatOpenAI(
    azure_deployment='azure/my_deployment',
    azure_endpoint='https://my_region.api.cognitive.microsoft.com/openai/deployments/my_deployment/chat/completions?api-version=some_version',
    api_version='some_version',
    api_key='my_key'
)