Why is returnDirect for Custom Tools of multi-action agents not supported?

[niklas-lohmann]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { DynamicTool } from "@langchain/core/tools";
import { ChatOpenAI } from "@langchain/openai";
import { AgentExecutor, createOpenAIFunctionsAgent } from "langchain/agents";
import { pull } from "langchain/hub";
import { customCallbackHandler } from "./callback-handler.js";

const questionOfLife = new DynamicTool({
  name: "Question_of_Life",
  description:
    "Rufe dies auf, um zu erfahren wie die endgültige Frage nach dem Leben, dem Universum und dem ganzen Rest beantwortet wird.",
  returnDirect: true,
  func: async () => "42"
});

const tools = [questionOfLife];

// Get the prompt to use - you can modify this!
// If you want to see the prompt in full, you can at:
// https://smith.langchain.com/hub/hwchase17/openai-functions-agent
const agentPrompt = await pull<ChatPromptTemplate>("hwchase17/openai-functions-agent");

const agentModel = new ChatOpenAI({
  modelName: "gpt-3.5-turbo-1106",
  temperature: 0
});

const agent = await createOpenAIFunctionsAgent({
  llm: agentModel,
  tools,
  prompt: agentPrompt
});

const agentExecutor = new AgentExecutor({
  agent,
  tools,
  verbose: false
});

class AgentService {
  async askLlm(msg: string) {
    return agentExecutor.invoke(
      { input: msg },
      {
        callbacks: [customCallbackHandler]
      }
    );
  }
}

export const agentService = new AgentService();

Description

I am trying to use OpenAI functions in combination with custom tools. For some use cases I want the output of a custom tool not to be parsed again by the LLM. I found the returnDirect option for custom tools in the documentation, but when I use the option, it returns the Error Tool with return direct ${tool.name} not supported for multi-action agent.. What is the reason why no direct return can be used for Open AI Function Agents?

System Info

[email protected]
Windows
[email protected]

[dosubot[bot]]

Hey @niklas-lohmann, nice to see you back here! Hope you're doing well.

The error you're encountering, "Tool with return direct ${tool.name} not supported for multi-action agent," is due to a specific check within the AgentExecutor constructor in the LangChain framework. This constructor checks if the agent being used is a "multi-action" type and if any of the tools specified for use with this agent have the returnDirect property set to true. If both conditions are met, the error is thrown.

Here's the relevant code snippet from the AgentExecutor constructor:

if (this.agent._agentActionType() === "multi") {
  for (const tool of this.tools) {
    if (tool.returnDirect) {
      throw new Error(
        `Tool with return direct ${tool.name} not supported for multi-action agent.`
      );
    }
  }
}

This design decision is likely made to ensure that the tools used with multi-action agents adhere to certain expectations regarding their output handling, which might not be compatible with the direct return mechanism.

The returnDirect property in the custom tools of the LangChain framework is used to determine whether the response from the chain should be the direct result of the graph query. When returnDirect is set to true, the chain directly returns the raw output of the graph query without any additional processing or formatting.

Unfortunately, this means that if you're using a multi-action agent, you won't be able to use the returnDirect property with your custom tools. You might need to find a different way to achieve your desired output. If you need further assistance, please provide more details about your use case and I'll do my best to help you find a solution.

You can find more details about this in the langchain/src/agents/executor.ts and examples/src/chains/graph_db_return_direct.ts files in the LangChain repository.

Sources

• langchain/src/agents/executor.ts
• examples/src/chains/graph_db_return_direct.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jacoblee93]

This sounds like an oversight - could you open an issue or (if you're feeling brave!) take a stab at a PR?

[niklas-lohmann]

I opened a PR (#4643). Thanks!

[zAlweNy26]

Hi guys, sorry to reopening this discussion, but I'm still getting this error even after updating:

• [email protected]
• @langchain/[email protected]

Am I doing something wrong?
This is my snippet of code

const agent = RunnablePassthrough.assign({
	agent_scratchpad: x => generatedScratchpad((x.intermediateSteps ?? []) as AgentStep[]),
}).pipe(prompt).pipe(this.verboseRunnable).pipe(currentLLM).pipe(new ProceduresOutputParser())

const agentExecutor = AgentExecutor.fromAgentAndTools({
	agent,
	tools,
	returnIntermediateSteps: true,
	maxIterations: 5,
})

that throws the error:

langchainjs/langchain/src/agents/executor.ts

Line 433 in ed1f3fb

`Tool with return direct ${tool.name} not supported for multi-action agent.`

[varunmehra5]

Still getting this error on langchain==0.2.5

[zAlweNy26]

Me too :(

[benm5678]

Did anyone find a solution? Since functions seem to be deprecated trying to switch to the new tools param.

[zAlweNy26]

Which one are you talking about?

[manish-chi]

Hi, I have a use case - I query for "add a Pizza to cart" the langchain understands the intent is "order" and adds the pizza to the cart!

To add the pizza in "addToCart" tool, it queries mongoDb database and returns me list of Pizza to get specific pizza. Now the results from the DB have image, name, price which I want to send it to my bot for displaying in cards but I am just getting a agent response as output with no DB list of Objects which I am sending from Tool.

  return new DynamicStructuredTool({
    name: "addToCart",
    description:
      "Adds one or more food items to the cart. Trigger this when the user says things like 'add', 'order', 'want', 'get', or 'have' followed by a dish name.",
    schema: z.object({
      items: z
        .array(
          z.object({
            dish: z.string().describe("Name of the food dish"),
            quantity: z.number().min(1).describe("how many units of dish"),
          })
        )
        .describe("List of dishes and their quantities"),
    }),
    returnType: AddToCartOutputSchema,
    returnDirect: true,
    func: async ({ items }, config) => {
      try {
        let messagesAdded = [];
        let cart = [];
        let foundItems = [];

        const rawCart = await redis.get(`cart:${config.metadata.sessionId}`);
        if (rawCart) {
          cart = typeof rawCart == "string" ? JSON.parse(rawCart) : rawCart;
        }

        for (let item of items) {
          let foodItem = await foodModel
            .findOne({
              name: { $regex: new RegExp(`^${item.dish}$`, "i") },
            })
            .select("name price_in_INR image")
            .lean();

          if (!foodItem) {
            messagesAdded.push(`${item.name} is not served here.`);
            continue;
          }

          foundItems.push(foodItem);

          const itemPrice = foodItem.price_in_INR;

          foodItem.price = item.quantity * itemPrice;

          let existing = cart.find(
            (c) => c.name.toLowerCase() === item.dish.toLowerCase()
          );

          if (existing) {
            existing.quanity += item.quantity;
            cart.quantity = existing.quanity;
          } else {
            foodItem.quantity = item.quantity;
            cart.push(foodItem);
          }
        }

        // Build summary
        let summary = "🛒 Here's your updated cart:\n";
        let totalItems = 0;

        cart.forEach((item, idx) => {
          summary += `${idx + 1}. ${item.quantity} x ${item.name} = ${
            item.price
          }🍽️\n`;
          totalItems += 1;
        });

        console.log(cart);

        await redis.set(
          `cart:${config.metadata.sessionId}`,
          JSON.stringify(cart)
        );

        summary += `❌${messagesAdded.join(
          ","
        )}\n\n✅ Total items: ${totalItems}\n🧾 You can proceed to checkout or add more items?`;

        return JSON.stringify({ summary: summary, cart_items: foundItems });
      } catch (err) {
        console.log(err);
      }
    },
  });
}```

I tried to add "returnDirect:true" but it seems that doesn't support multiagent. I just don't understand how do I implement this solution.