Agent with tools and return structured output

[zirkelc]

I have an Agent with one dynamic structured tool and I want the agent to return a structured response. I followed the example on the documentation to implement the agent. However, the agent doesn't always return a structured response.

Here is my code example from the documentation. I extended the response schema with `answer`, `classification` and `sentiment`. The websearch tool also requires structured input.

import { zodToJsonSchema } from 'zod-to-json-schema';
import { z } from 'zod';
import { type BaseMessage, AIMessage, FunctionMessage, type AgentFinish, type AgentStep } from 'langchain/schema';
import { RunnableSequence } from 'langchain/runnables';
import { ChatPromptTemplate, MessagesPlaceholder } from 'langchain/prompts';
import { ChatOpenAI } from '@langchain/openai';
import { AgentExecutor } from 'langchain/agents';
import { DynamicTool, DynamicStructuredTool } from '@langchain/core/tools';
import type { FunctionsAgentAction } from 'langchain/agents/openai/output_parser';
import { convertToOpenAIFunction } from '@langchain/core/utils/function_calling';
import { TavilySearchAPIRetriever } from '@langchain/community/retrievers/tavily_search_api';

const OPENAI_API_KEY = 'sk-...';
const TAVILY_API_KEY = 'tvly-...';

  const llm = new ChatOpenAI({
    modelName: 'gpt-4-0125-preview',
    openAIApiKey: OPENAI_API_KEY,
  });

  const taviliyRetriever = new TavilySearchAPIRetriever({ apiKey: TAVILY_API_KEY });

  const searchTool = new DynamicStructuredTool({
    name: 'web-search-tool',
    description: 'Tool for getting the latest information from the web',
    schema: z.object({
      question: z.string().describe('The current question'),
      history: z.string().describe('The chat history'),
      summary: z.string().describe('The summary of the chat'),
      random: z.number().describe('A random number'),
    }),
    func: async (input: any, runManager) => {
      console.log('input', input);
      const docs = await taviliyRetriever.invoke(input.question, runManager?.getChild());
      return docs.map((doc) => doc.pageContent).join('\n-----\n');
    },
    // func: async (searchQuery: string, runManager) => {
    //   const retriever = new TavilySearchAPIRetriever();
    //   const docs = await retriever.invoke(searchQuery, runManager?.getChild());
    //   return docs.map((doc) => doc.pageContent).join('\n-----\n');
    // },
  });

  const prompt = ChatPromptTemplate.fromMessages([
    [
      'system',
      `You are a helpful assistant. You must always call one of the provided tools.

Chat history:
{history}
    `,
    ],
    ['user', '{question}'],
    new MessagesPlaceholder('agent_scratchpad'),
  ]);

  const responseSchema = z.object({
    answer: z.string().describe('The final answer to return to the user'),
    classification: z
      .union([
        z.literal('question'),
        z.literal('complaint'),
        z.literal('compliment'),
        z.literal('suggestion'),
        z.literal('other'),
      ])
      .describe('Type of message: question, complaint, compliment, suggestion, other'),
    sentiment: z
      .union([z.literal('positive'), z.literal('negative'), z.literal('neutral')])
      .describe('Sentiment of message: positive, negative, neutral, unknown'),
  });

  const responseOpenAIFunction = {
    name: 'response',
    description: 'Return the response to the user',
    parameters: zodToJsonSchema(responseSchema),
  };

  const structuredOutputParser = (message: AIMessage): FunctionsAgentAction | AgentFinish => {
    if (message.content && typeof message.content !== 'string') {
      throw new Error('This agent cannot parse non-string model responses.');
    }
    if (message.additional_kwargs.function_call) {
      const { function_call } = message.additional_kwargs;
      try {
        const toolInput = function_call.arguments ? JSON.parse(function_call.arguments) : {};
        // If the function call name is `response` then we know it's used our final
        // response function and can return an instance of `AgentFinish`
        if (function_call.name === 'response') {
          return { returnValues: { ...toolInput }, log: message.content };
        }
        return {
          tool: function_call.name,
          toolInput,
          log: `Invoking "${function_call.name}" with ${function_call.arguments ?? '{}'}\n${message.content}`,
          messageLog: [message],
        };
      } catch (error) {
        throw new Error(
          `Failed to parse function arguments from chat model response. Text: "${function_call.arguments}". ${error}`,
        );
      }
    } else {
      return {
        returnValues: { output: message.content },
        log: message.content,
      };
    }
  };

  const formatChatHistory = (
    messages?: Array<{
      body: string;
      role: string;
    }>,
  ): string => {
    if (!messages || messages.length === 0) return '';

    return messages
      .map((message): string => {
        if (typeof message === 'string') return message;

        return message.role === 'human'
          ? `Human: ${message.body}`
          : message.role === 'ai'
            ? `Assistant: ${message.body}`
            : `${message.body}`;
      })
      .join('\n');
  };

  const formatAgentSteps = (steps: AgentStep[]): BaseMessage[] =>
    steps.flatMap(({ action, observation }) => {
      if ('messageLog' in action && action.messageLog !== undefined) {
        const log = action.messageLog as BaseMessage[];
        return log.concat(new FunctionMessage(observation, action.tool));
      }
      return [new AIMessage(action.log)];
    });

  const llmWithTools = llm.bind({
    functions: [convertToOpenAIFunction(searchTool), responseOpenAIFunction],
  });
  /** Create the runnable */
  const runnableAgent = RunnableSequence.from<{
    question: string;
    history: string;
    steps: Array<AgentStep>;
  }>([
    {
      question: (i) => i.question,
      history: (i) => i.history,
      agent_scratchpad: (i) => formatAgentSteps(i.steps),
    },
    prompt,
    llmWithTools,
    structuredOutputParser,
  ]);

  const executor = AgentExecutor.fromAgentAndTools({
    agent: runnableAgent,
    tools: [searchTool],
  });

  const history: Array<{
    body: string;
    role: string;
  }> = [];

  let question = `what is the current weather in honolulu?`;

  /** Call invoke on the agent */
  let result = await executor.invoke({
    question,
    history: formatChatHistory(history),
  });

  console.log(result);
  history.push({ role: 'human', body: question });
  history.push({ role: 'ai', body: result.output ?? result.answer });

  question = `and in miami?`;

  result = await executor.invoke({
    question,
    history: formatChatHistory(history),
  });

  console.log(result);
  history.push({ role: 'human', body: question });
  history.push({ role: 'ai', body: result.output ?? result.answer });

  question = `great, what about in new york?`;

  result = await executor.invoke({
    question,
    history: formatChatHistory(history),
  });

  console.log(result);
  history.push({ role: 'human', body: question });
  history.push({ role: 'ai', body: result.output ?? result.answer });

I get the following results.

First response to what is the current weather in honolulu?

{
  question: 'what is the current weather in honolulu?',
  history: '',
  answer: 'The current weather in Honolulu is overcast with a temperature of 21°C (69.8°F). The humidity is at 78%, with a wind speed of 6.9 mph (11.2 kph) coming from the east. There is a slight precipitation of 0.17 mm and the visibility is 16 km (9 miles).',
  classification: 'question',
  sentiment: 'neutral'
}

Second response to and in miami?

{
  question: 'and in miami?',
  history: 'Human: what is the current weather in honolulu?\n' +
    'Assistant: The current weather in Honolulu is overcast with a temperature of 21°C (69.8°F). The humidity is at 78%, with a wind speed of 6.9 mph (11.2 kph) coming from the east. There is a slight precipitation of 0.17 mm and the visibility is 16 km (9 miles).',
  output: 'The current weather in Miami, Florida, is sunny with a temperature of 13.3°C (55.9°F). The wind is coming from the NNW at 6.1 kph (3.8 mph). The humidity is at 77%, and there is no precipitation. The feels-like temperature is 11.7°C (53.1°F), and the visibility is 16 km (9 miles).'
}

Third response to great, what about in new york?

{
  question: 'great, what about in new york?',
  history: 'Human: what is the current weather in honolulu?\n' +
    'Assistant: The current weather in Honolulu is overcast with a temperature of 21°C (69.8°F). The humidity is at 78%, with a wind speed of 6.9 mph (11.2 kph) coming from the east. There is a slight precipitation of 0.17 mm and the visibility is 16 km (9 miles).\n' +
    'Human: and in miami?\n' +
    'Assistant: The current weather in Miami, Florida, is sunny with a temperature of 13.3°C (55.9°F). The wind is coming from the NNW at 6.1 kph (3.8 mph). The humidity is at 77%, and there is no precipitation. The feels-like temperature is 11.7°C (53.1°F), and the visibility is 16 km (9 miles).',
  answer: 'The current weather in New York is partly cloudy with a temperature of 1.7°C (35.1°F). The wind is coming from the south at 3.6 kph (2.2 mph). The humidity is at 69%, and there is no precipitation. The feels-like temperature is -0.3°C (31.6°F), and the visibility is 16 km (9 miles).',
  classification: 'question',
  sentiment: 'neutral'
}

As you can see, the first and third responses are structured with answer, classification and sentiment, while the second response only contains the default response key output.

Is there a way to consistently get a structured response?

[dosubot[bot]]

🤖

Hey @zirkelc, great to see you back here! Hope you're doing well.

Based on the information you've provided and the context from the langchainjs repository, it seems like the issue you're experiencing might be due to the fact that the output parser for the StructuredChatAgent is not always guaranteed to return a structured response.

The output parser used in the StructuredChatAgent is the StructuredChatOutputParserWithRetries class. This class attempts to parse the output from the language model into a structured format, but if it fails to do so, it will return the raw output from the language model. This is evident in the getDefaultOutputParser method of the StructuredChatAgent class:

static getDefaultOutputParser(
  fields?: OutputParserArgs & {
    toolNames: string[];
  }
) {
  if (fields?.llm) {
    return StructuredChatOutputParserWithRetries.fromLLM(fields.llm, {
      toolNames: fields.toolNames,
    });
  }
  return new StructuredChatOutputParserWithRetries({
    toolNames: fields?.toolNames,
  });
}

In this method, if an LLM (Language Learning Model) is provided, it creates an output parser with retry logic from the LLM. If the LLM is not provided, it creates a new instance of StructuredChatOutputParserWithRetries with the provided tool names. In both cases, the output parser is an instance of StructuredChatOutputParserWithRetries, which may not always return a structured response.

If you want to ensure a structured response, you might need to implement additional error handling or fallback logic in your application. You can find more details about this in the langchain/src/agents/structured_chat/index.ts file.

Regarding the StructuredChatOutputParserWithRetries class, it is a wrapper around the StructuredChatOutputParser and OutputFixingParser classes. It extends the AgentActionOutputParser class and allows for retrying the output parsing using the OutputFixingParser if it is provided. You can find more details about this in the langchain/src/agents/structured_chat/outputParser.ts file.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• StructuredOutputParser Not working?
• langchain/src/agents/structured_chat/index.ts
• langchain/src/agents/structured_chat/outputParser.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[zirkelc]

@dosu-bot I'm using OpenAI Tools Agent and not the StructuredChatAgent

[dosubot[bot]]

🤖

It sounds like you're using the OpenAI Tools Agent instead of the StructuredChatAgent. How can I assist you with that?

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[simontam]

I'm trying to do something similar but not having any luck. You're probably farther along than I am in that you got it to partially work. I can't even get that far. I think you're using Langchain JS whereas I'm using Langchain Python. What have you done thus far or have you solved it?

[zirkelc]

I didn't really continue on this topic for now, because I didn't really get the feeling that the latest GPT4 model is ready for a multi-step agent. Or in other words, coding the agent from hand with zero-shot prompts and structured outputs returned muich better results that the agent version.

If you want to see the code how I have implemented it, it is in my original post hidden behind the toggle: