Does @langchain/openai AzureChatOpenAI respect the retry-after header when being rate-limited?

[JonathanVelkeneers]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

This is not the actual code i'm trying to run, but just an example to attempt to hit the rate limit. (RPM/TPM)

import { AzureChatOpenAI } from '@langchain/openai';

const generationChatModel = new AzureChatOpenAI({
  azureOpenAIApiKey: process.env.GENERATION_AZURE_OPENAI_API_KEY,
  azureOpenAIApiInstanceName: process.env.GENERATION_AZURE_OPENAI_API_INSTANCE_NAME,
  azureOpenAIApiDeploymentName: process.env.GENERATION_AZURE_OPENAI_API_DEPLOYMENT_NAME,

  model: 'gpt-4o',
  maxTokens: 128_000,
});

const systemPrompt = "...";
const text = "...";

// Example to bump into rate-limit azure openAI ChatCompletions_Create
for (let i = 0; i < 150; i++) {
    const result = await generationChatModel.invoke([
      ['system', systemPrompt],
      ['human', text],
    ]);
    console.log(result.content);
}

Description

I have a code-flow that often bumps in to Azure OpenAI rate limits. This itself is not a problem i'm trying to fix.

I am however curious if Langchain.JS uses the Azure provided 'retry-after' header to set it's retry-logic's timeout?
Based on my logs it seems as if the code just bruteforces it with some arbitrary timeout

System Info

[email protected]
macos
node 20.x
npm 10.8.2

[dosubot[bot]]

Hey @JonathanVelkeneers! I'm here to assist you with any bugs, questions, or contributions you may have. Let's work together to resolve your concerns.

The AzureChatOpenAI class in the LangChain JS repository does not specifically check for the retry-after header in the response from the Azure OpenAI API ^[1].

To continue talking to Dosu, mention @dosu.

[JonathanVelkeneers]

@dosu Can this be requested through a feature request?

[dosubot[bot]]

Yes, you can request this feature by opening a feature request in the issues section of this repository. Make sure to provide a detailed description of the feature and its benefits. This will help the maintainers understand the importance and prioritize it accordingly.

To continue talking to Dosu, mention @dosu.

[JonathanVelkeneers]

@dosu it seems your previous answer references the deprecated @langchain/azure-openai version of AzureChatOpenAI.
Can you formulate an answer with code from '@langchain/openai' ?

[dosubot[bot]]

The AzureChatOpenAI class in the @langchain/openai package does not specifically check for the retry-after header in the response from the Azure OpenAI API ^[1][2].

To continue talking to Dosu, mention @dosu.