Streaming OpenAI Function Call output in LangChainStreaming

[nikohann]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { NextRequest, NextResponse } from "next/server";
import { StreamingTextResponse, LangChainStream } from "ai";
import { ChatOpenAI } from "@langchain/openai";
import { PromptTemplate } from "@langchain/core/prompts";
import { SupabaseVectorStore } from "@langchain/community/vectorstores/supabase";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { zodToJsonSchema } from "zod-to-json-schema";
import { JsonOutputFunctionsParser } from "langchain/output_parsers";
import { StringOutputParser } from "@langchain/core/output_parsers";
import { z } from "zod";
import { RunnableSequence } from "@langchain/core/runnables";
import { VoyageEmbeddings } from "@langchain/community/embeddings/voyage";
import { createClient } from "@/utils/supabase/server";
import { createRetrievalChain } from "langchain/chains/retrieval";
import { createStuffDocumentsChain } from "langchain/chains/combine_documents";

export const runtime = "edge";

const prompt = ChatPromptTemplate.fromMessages([
    [
        "system",
        `Imagine that you are helpful bot. You are answering to users' questions.
        
        <context>
            {context}
        </context>`,
    ],
    ["human", "{input}"],
]);


const documentTemplate = PromptTemplate.fromTemplate("{title}: {page_content}");

export async function POST(req: NextRequest) {
    try {

        const schema = z.object({
            answer: z
                .string()
                .describe(
                    "Answer to user question."
                ),
            citations: z
                .array(z.string())
                .describe(
                    "Used citations (titles) in generated answer."
                ),
        });

        const modelParams = {
            functions: [
                {
                    name: "cited_answer",
                    description: "Answer to user's question and use citations",
                    parameters: zodToJsonSchema(schema),
                },
            ],
            function_call: { name: "cited_answer" },
        };

        const llm = new ChatOpenAI({
            modelName: "gpt-4-0125-preview",
            streaming: true,
            temperature: 0.2,
        }).bind(modelParams);

        const embeddings = new VoyageEmbeddings({
            apiKey: process.env.VOYAGE_API_KEY,
            modelName: "voyage-large-2",
        });

        const client = createClient();

        const vectorstore = new SupabaseVectorStore(embeddings, {
            client,
            tableName: "documents",
            queryName: "match_documents",
        });

        const retriever = vectorstore.asRetriever();

        const outputParser = new JsonOutputFunctionsParser();

        const combineDocsChain = await createStuffDocumentsChain({ llm: llm, prompt: prompt, outputParser: outputParser, documentPrompt: documentTemplate })

        const retrievalChain = await createRetrievalChain({
            retriever,
            combineDocsChain
        });

        //This part is not working
        //StringOutputParser returns undefined chunks and therefore StreamingTextResponse not working
        const chain = RunnableSequence.from([
            retrievalChain,
            new StringOutputParser(),
        ])

        const { stream, handlers } = LangChainStream();

        chain.invoke({
            input: "What is a patent right?"
        }, {
            callbacks: [handlers]
        })

        /*
        //This part is working and chunks' are printing in console. If you remove console.log part, then StreamingTextResponse will work.
        //Remove 1) chain.invoke ...,
        const stream = await retrievalChain.stream({
            input: "What is a patent right?"
        })

        for await (const chunk of stream) {
            console.log(chunk)
        }
        */

        return new StreamingTextResponse(stream);
    } catch (e: any) {
        return NextResponse.json({ error: e.message }, { status: e.status ?? 500 });
    }
}

Description

While working on my project, I faced an issue with the StringOutputParser. It returns undefined as output for all processed input. I attempted to resolve the problem by creating a CustomOutputParser, but unfortunately, I encountered the same issue. Even with StreamingTextResponse, everything works normally when I use a pure LLM model without function calling.

The main issue concerns stream OpenAI functions output to the frontend.

System Info

Langchain v0.1.27
Macbook Air 2020 M1
Node v18.19.1
NPM v10.2.4

[dosubot[bot]]

Hey @nikohann, great to see you diving into new challenges! Hope everything's been going well since our last interaction.

Based on the code you've provided and the context, it seems like the issue you're facing is related to the StringOutputParser returning undefined for all processed input when streaming OpenAI function output to the frontend in LangChain JS.

The StringOutputParser is designed to parse the output from a language model (LLM) call into a string. It extends the BaseTransformOutputParser class, indicating it is part of a broader framework for handling different types of output parsing. The primary functionality of this parser is to take the raw string output from an LLM call and return it directly without modification. You can find more details about its functionality in the LangChain JS library.

The JsonOutputFunctionsParser in the LangChain JS library is designed to parse the output of a Language Model (LLM) into a JSON object. It extends the BaseCumulativeTransformOutputParser class and provides functionality to handle JSON outputs specifically. More details about its functionality can be found in the LangChain JS library.

The LangChainStream function, as demonstrated in the provided code snippet from the examples/src/chains/llm_chain_stream.ts file, is not explicitly named in the snippet. However, the functionality related to streaming in the LangChain JS library can be inferred from the context of using the LLMChain class in conjunction with the OpenAI model configured for streaming. More details about its functionality can be found in the LangChain JS library.

Given the correct usage of StringOutputParser and ChatOpenAI as shown in the provided code, and assuming there are no internal bugs within the @langchain/openai and @langchain/core libraries themselves, the issue might stem from one of the following:

1. Improper Stream Handling: If the StringOutputParser is not correctly handling the stream of data from the ChatOpenAI model, it might not be parsing the incoming data correctly, leading to undefined outputs. This could happen if the parser expects a certain format or delimiter in the stream that is not present.

2. Configuration Issue: The configuration passed to ChatOpenAI (e.g., { temperature: 0 }) might be influencing the output in a way that is not compatible with the StringOutputParser. For example, a temperature of 0 makes the model's responses deterministic, but this should not directly cause an issue with parsing unless the parser has specific expectations.

3. Async/Await Misuse: The asynchronous iteration over the stream (for await (const chunk of stream)) is correct, but if there's any surrounding asynchronous context or handling that's not shown here that is improperly implemented, it could lead to timing issues or missed data chunks.

4. Library Version Mismatch or Bug: If you're using versions of @langchain/openai and @langchain/core that are not compatible with each other, or if there's a bug in the versions you're using, that could lead to unexpected behavior such as receiving undefined for all processed input.

Without more information on the exact output you're receiving (e.g., if undefined is being logged for each chunk or if the issue occurs after some processing not shown here), or without details on the versions of the libraries used and their compatibility, it's challenging to pinpoint the exact cause. It's recommended to ensure all libraries are up-to-date and to review the documentation for StringOutputParser and ChatOpenAI for any specific configuration requirements or known issues.

I hope this helps! If you have any more information or if there's anything else you'd like me to look into, please let me know.

Sources

• LangchainJs Streaming with OpenAIFunctions Agent
• examples/src/prompts/string_output_parser.ts
• langchain-core/src/output_parsers/string.ts
• langchain-core/src/output_parsers/json.ts
• examples/src/chains/llm_chain_stream.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nikohann]

@dosu-bot How to handle the stream data from the ChatOpenAI model, when the stream data is under function_call?

[dosubot[bot]]

Based on the context provided, it seems like you're trying to handle streaming data from the ChatOpenAI model using the StringOutputParser in the LangChain JS library. However, you're encountering an issue where the StringOutputParser is returning undefined for all processed input.

A similar issue was found in the LangChain JS repository, as mentioned in the issue Is it possible to add streaming to a ConversationChain?. The suggested solution in that issue was to use a RunnableSequence without the StringOutputParser and handle the new tokens from the stream in a callback function, like so:

const chain = RunnableSequence.from([
    {
        question: (input: { question: string}) =>
        input.question,
    },
    prompt,
    model,
]);

const stream = await chain
.stream({ question: "hello" }, {
    callbacks: [{
        handleLLMNewToken(token: string) {
            console.log({ token });
        }
    }]
});
for await (const chunk of stream);

In this example, the handleLLMNewToken function is correctly capturing and logging each token received from the ChatOpenAI model. The tokens are received as part of the stream and logged individually. This is the expected behavior when working with streaming responses, where the data is received and processed in chunks (in this case, tokens) rather than all at once.

If you're experiencing issues with StringOutputParser returning undefined, it's important to ensure that the parser is correctly implemented and integrated with your streaming data handling logic. However, based on the provided context, the issue seems unrelated to the streaming data handling itself, as the tokens are being correctly received and logged. The problem might lie elsewhere in your implementation or usage of StringOutputParser.

The StringOutputParser in the LangChain JS library is designed to parse the result from a Large Language Model (LLM) call into a string, specifically the top likely string output. It extends the BaseTransformOutputParser class and implements a parse method that simply returns the input string as a promise, indicating that it does not modify the input string in any way. This class is intended for use in a chain of operations where an LLM's output needs to be treated as a simple string, for example, in a sequence that involves generating text based on a prompt and then processing that text further.

Ensure that StringOutputParser is correctly set up to parse the incoming stream data and that it is not being called prematurely before the stream has been fully processed. Without specific details on how StringOutputParser is implemented or integrated into your code, it's challenging to provide a more targeted solution. However, the provided example demonstrates correct handling of streaming data from the ChatOpenAI model.

For more details, you can refer to the LangChain JS library's source code, specifically the examples/src/models/chat/chat_streaming.ts and langchain-core/src/output_parsers/string.ts files.

Sources

• Is it possible to add streaming to a ConversationChain?
• examples/src/models/chat/chat_streaming.ts
• langchain-core/src/output_parsers/string.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.