ChatOpenAI with Azure OpenAI service returns a short response.

[kj-itk]

Hi everyone.

I got a sveltekit app with langchain and ai vercel and it works ok but the response is short like 300 tokens. I tried to add maxTokens to ChatOpenAI. The response is the same. Anyone faced the same issue?

import type { Message } from 'ai';
import { StreamingTextResponse } from 'ai';
import type { RequestHandler } from './$types';
import { ChatOpenAI } from 'langchain/chat_models/openai';
import { BytesOutputParser } from 'langchain/schema/output_parser';
import { PromptTemplate } from 'langchain/prompts';

import {
	OPENAI_API_KEY,
	OPENAI_API_VERSION,
	OPENAI_AZURE_DEPLOYMENT_NAME,
	OPENAI_AZURE_INSTANCE_NAME
} from '$env/static/private';

const formatMessage = (message: Message) => {
	return `${message.role}: ${message.content}`;
};

const TEMPLATE = `=You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.
   
  Current conversation:
  {chat_history}
   
  User: {input}
  AI:`;

/*
 * This handler initializes and calls a simple chain with a prompt,
 * chat model, and output parser. See the docs for more information:
 *
 * https://js.langchain.com/docs/guides/expression_language/cookbook#prompttemplate--llm--outputparser
 */
export const POST: RequestHandler = async ({ request }) => {
	const body = await request.json();
	const messages = body.messages ?? [];
	const formattedPreviousMessages = messages.slice(0, -1).map(formatMessage);
	const currentMessageContent = messages[messages.length - 1].content;

	const prompt = PromptTemplate.fromTemplate(TEMPLATE);
	/**
	 * See a full list of supported models at:
	 * https://js.langchain.com/docs/modules/model_io/models/
	 */

	const model = new ChatOpenAI({
		temperature: 1,
		azureOpenAIApiKey: OPENAI_API_KEY,
		azureOpenAIApiVersion: OPENAI_API_VERSION,
		azureOpenAIApiDeploymentName: OPENAI_AZURE_DEPLOYMENT_NAME,
		azureOpenAIApiInstanceName: OPENAI_AZURE_INSTANCE_NAME
	});

	/**
	 * Chat models stream message chunks rather than bytes, so this
	 * output parser handles serialization and encoding.
	 */
	const outputParser = new BytesOutputParser();

	/*
	 * Can also initialize as:
	 *
	 * import { RunnableSequence } from "langchain/schema/runnable";
	 * const chain = RunnableSequence.from([prompt, model, outputParser]);
	 */
	const chain = prompt.pipe(model).pipe(outputParser);

	const stream = await chain.stream({
		chat_history: formattedPreviousMessages.join('\n'),
		input: currentMessageContent
	});

	return new StreamingTextResponse(stream);
};

[dosubot[bot]]

🤖

Hello kj-itk,

Based on the code you've shared, it seems like you're not setting the maxTokens parameter in the ChatOpenAI model. This parameter controls the maximum number of tokens that the model will generate in response to the input. If you want to increase the number of tokens in the response, you need to increase the value of maxTokens.

Here's how you can modify the code:

const model = new ChatOpenAI({
	temperature: 1,
	maxTokens: 500, // Increase the maxTokens value here
	azureOpenAIApiKey: OPENAI_API_KEY,
	azureOpenAIApiVersion: OPENAI_API_VERSION,
	azureOpenAIApiDeploymentName: OPENAI_AZURE_DEPLOYMENT_NAME,
	azureOpenAIApiInstanceName: OPENAI_AZURE_INSTANCE_NAME
});

In this updated code, the maxTokens parameter is set to 500, which means the model will generate up to 500 tokens in response to the input. Please note that increasing the maxTokens value will also increase the time it takes for the model to generate a response.

I found a similar issue in the langchainjs repository: How can i get longer responses?. However, it was closed without any comments or solution.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• How can i get longer responses?
• examples/src/llms/openai-chat.ts
• examples/src/models/chat/chat_quick_start.ts
• examples/src/models/chat/chat_streaming.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kj-itk]

I tried it and set it to 2024 but the output is shorter but thank u! It ain't that critical!