Please Help fixing issue with creating RAG with prompt, chat_history and tools

[Originn]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const streamingModel = new ChatOpenAI({
    streaming: true,
    modelName: MODEL_NAME,
    temperature: TEMPRATURE,
    verbose: false,
    modelKwargs: {
      seed: 1,
    },
  });

  const addTool = new DynamicStructuredTool({
    name: "add",
    description: "Adds a and b.",
    schema: z.object({
      a: z.number(),
      b: z.number(),
    }),
    func: async ({ a, b }) => {
      return `${a + b}`;
    },
  });
  
  const multiplyTool = new DynamicStructuredTool({
    name: "multiply",
    description: "Can perform multiplications operations.",
    schema: z.object({
      a: z.number(),
      b: z.number(),
    }),
    func: async ({ a, b }) => {
      console.log(`Multiply tool called with input: { a: ${a}, b: ${b} }`);
      const result = a * b * a;
      return result.toString();
    },
  });
  
  const tools = [addTool, multiplyTool];

  const llmWithTools = streamingModel.bindTools(tools, {
    tool_choice: "any",
  })

const ragChain = await createRetrievalChain({
        retriever: await createHistoryAwareRetriever({
          llm: nonStreamingModel,
          retriever: customRetriever,
          rephrasePrompt: contextualizeQPrompt,
        }),
        combineDocsChain: await createStuffDocumentsChain({
          llm: llmWithTools,
          prompt: qaPrompt,
        }),
      });

      const chatHistory = await memory.loadMemoryVariables({});

      const response = await ragChain.invoke({
        input,
        chat_history: chatHistory.chat_history
      }, {
        callbacks: [
          {
            handleLLMNewToken: (token) => {
              console.log(`Received token: ${token}`);
              onTokenStream(token);
            },
            handleLLMEnd: async (output) => {
              console.log('LLM End Output:', JSON.stringify(output, null, 2));
              
              const generation = output.generations[0][0];
              if ('message' in generation) {
                const chatGeneration = generation as ChatGeneration;
                if (chatGeneration.message.additional_kwargs?.tool_calls) {
                  for (const toolCall of chatGeneration.message.additional_kwargs.tool_calls) {
                    console.log('Tool Call:', JSON.stringify(toolCall, null, 2));
                    
                    const functionName = toolCall.function.name;
                    const args = JSON.parse(toolCall.function.arguments);
                    console.log(`Function called: ${functionName}`);
                    console.log('Arguments:', args);
                    
                    // Execute the tool and stream only its result
                    const tool = tools.find(t => t.name === functionName);
                    if (tool) {
                      try {
                        const result = await tool.invoke(args);
                        // Stream only the result
                        onTokenStream(result.toString());
                      } catch (error) {
                        console.error(`Error executing tool ${functionName}:`, error);
                      }
                    } else {
                      console.warn(`Tool ${functionName} not found`);
                    }
                  }
                }
              }
            },
            handleToolStart: (toolStart) => {
              console.log(`Tool Start: ${toolStart.name}`);
              console.log(`Tool Input: ${JSON.stringify(toolStart)}`);
              // Add any additional logging or processing for tool start here
            },
          },
        ],
      });

Description

I am trying to incorporate tools into my RagChain and i have setup the above code.

It will work, but I have few issues:

I must enforce the tools

  const llmWithTools = streamingModel.bindTools(tools, {
    tool_choice: "any",
  })

if I don't do it and pass something like

const llmWithTools = streamingModel.bindTools(tools)

the LLMmodel would not call the Multiply tool if I ask how much is 12 times 12.

and

I have set up a handler to catch different callbacks but what I see is (when tools are enforced)

[llm/end] [1:chain:retrieval_chain > 8:chain:RunnableMap > 9:chain:RunnableSequence > 21:llm:ChatOpenAI] [879ms] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "",
        "generationInfo": {
          "prompt": 0,
          "completion": 0,
          "finish_reason": "tool_calls",
          "system_fingerprint": "fp_bc2a86f5f5"
        },
        "message": {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain_core",
            "messages",
            "AIMessageChunk"
          ],
          "kwargs": {
            "content": "",
            "additional_kwargs": {
              "tool_calls": [
                {
                  "index": 0,
                  "id": "call_8C4zv7USACD9Ul48u9DXN6t0",
                  "type": "function",
                  "function": {
                    "name": "multiply",
                    "arguments": "{\"a\":12,\"b\":12}"
                  }
                }
              ]
            },
            "response_metadata": {
              "estimatedTokenUsage": {
                "promptTokens": 1253,
                "completionTokens": 0,
                "totalTokens": 1253
              },
              "prompt": 0,
              "completion": 0,
              "finish_reason": "tool_calls",
              "system_fingerprint": "fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5"
            },
            "tool_call_chunks": [
              {
                "name": "multiply",
                "args": "{\"a\":12,\"b\":12}",
                "id": "call_8C4zv7USACD9Ul48u9DXN6t0",
                "index": 0,
                "type": "tool_call_chunk"
              }
            ],
            "id": "chatcmpl-9qws75ayWPQTkXaitkQbfraYq9UnY",
            "usage_metadata": {
              "input_tokens": 1312,
              "output_tokens": 15,
              "total_tokens": 1327
            },
            "tool_calls": [
              {
                "name": "multiply",
                "args": {
                  "a": 12,
                  "b": 12
                },
                "id": "call_8C4zv7USACD9Ul48u9DXN6t0",
                "type": "tool_call"
              }
            ],
            "invalid_tool_calls": []
          }
        }
      }
    ]
  ],
  "llmOutput": {
    "estimatedTokenUsage": {
      "promptTokens": 1253,
      "completionTokens": 0,
      "totalTokens": 1253
    }
  }
}

without showing any tool start even when I have added handleToolStart anywhere possible in my code

So my only option is to catch the tool call in the handleLLMEnd, as also it does not appear initially in the handleLLMStart

So I ask for help in:

1. Want to know if there is another way to call tools and using createRetrievalChain

2. understanding why the llmwithtools will not call the tools unless it is enforced.

regarding question 2 I have a hint in the logs of the callbacks as with tools not enforced I get

[chain/start] [1:chain:retrieval_chain > 8:chain:RunnableMap > 9:chain:RunnableSequence > 22:parser:StrOutputParser] Entering Chain run with input: {
  "lc": 1,
  "type": "constructor",
  "id": [
    "langchain_core",
    "messages",
    "AIMessageChunk"
  ],
  "kwargs": {
    "content": "The result of multiplying 12 by 12 is **144**.",
    "additional_kwargs": {},
    "response_metadata": {
      "estimatedTokenUsage": {
        "promptTokens": 1253,
        "completionTokens": 14,
        "totalTokens": 1267
      },
      "prompt": 0,
      "completion": 0,
      "finish_reason": "stop",
      "system_fingerprint": "fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518fp_4e2b2da518"
    },
    "tool_call_chunks": [],
    "id": "chatcmpl-9qwphGuyaWNdguIZLvhXofYYVnJGh",
    "usage_metadata": {
      "input_tokens": 1310,
      "output_tokens": 15,
      "total_tokens": 1325
    },
    "tool_calls": [],
    "invalid_tool_calls": []
  }
}

and with the tools enforced I get

[chain/start] [1:chain:retrieval_chain > 8:chain:RunnableMap > 9:chain:RunnableSequence > 22:parser:StrOutputParser] Entering Chain run with input: {
  "lc": 1,
  "type": "constructor",
  "id": [
    "langchain_core",
    "messages",
    "AIMessageChunk"
  ],
  "kwargs": {
    "content": "",
    "additional_kwargs": {
      "tool_calls": [
        {
          "index": 0,
          "id": "call_8C4zv7USACD9Ul48u9DXN6t0",
          "type": "function",
          "function": {
            "name": "multiply",
            "arguments": "{\"a\":12,\"b\":12}"
          }
        }
      ]
    },
    "response_metadata": {
      "estimatedTokenUsage": {
        "promptTokens": 1253,
        "completionTokens": 0,
        "totalTokens": 1253
      },
      "prompt": 0,
      "completion": 0,
      "finish_reason": "tool_calls",
      "system_fingerprint": "fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5fp_bc2a86f5f5"
    },
    "tool_call_chunks": [
      {
        "name": "multiply",
        "args": "{\"a\":12,\"b\":12}",
        "id": "call_8C4zv7USACD9Ul48u9DXN6t0",
        "index": 0,
        "type": "tool_call_chunk"
      }
    ],
    "id": "chatcmpl-9qws75ayWPQTkXaitkQbfraYq9UnY",
    "usage_metadata": {
      "input_tokens": 1312,
      "output_tokens": 15,
      "total_tokens": 1327
    },
    "tool_calls": [
      {
        "name": "multiply",
        "args": {
          "a": 12,
          "b": 12
        },
        "id": "call_8C4zv7USACD9Ul48u9DXN6t0",
        "type": "tool_call"
      }
    ],
    "invalid_tool_calls": []
  }
}

so the presence of

  "kwargs": {
    "content": "The result of multiplying 12 by 12 is **144**.",

will change if the tool is called or not, but I might be mistaking

System Info

[email protected]
"@langchain/core": "^0.2.18",
"@langchain/langgraph": "^0.0.31",
"@langchain/openai": "^0.2.5",

[dosubot[bot]]

To address your issues with creating a RAG (Retrieval-Augmented Generation) setup with prompt, chat history, and tools, here are some solutions and explanations:

1. Ensuring Tools Are Called Correctly

To ensure that tools are called correctly without enforcing them using tool_choice: "any", you can use the tool_choice: "auto" option. This allows the system to automatically select the appropriate tool based on the context and available tools.

Here's how you can modify your code:

const llmWithTools = streamingModel.bindTools(tools, {
  tool_choice: "auto",
});

2. Understanding Why Tools Are Not Called Unless Enforced

The issue you're facing where tools are not called unless enforced might be due to the way the model interprets the input and decides whether to use a tool. When tool_choice: "any" is used, it forces the model to consider all tools, but with tool_choice: "auto", the model should automatically select the appropriate tool based on the context.

3. Handling Callbacks for Tool Invocation

To ensure that your callback handlers capture all relevant events during tool invocation, follow these best practices:

• Implement All Relevant Methods: Make sure your CallbackHandler implements all the methods for the events you want to capture.
• Pass Callbacks Appropriately: Use the callbacks property to pass your handlers at the time of the request or in the constructor.
• Propagate Callbacks in Custom Chains: Ensure that you propagate request-time callbacks to any child objects.

Here is an example of how to set up a comprehensive callback handler:

class MyCallbackHandler {
  handleToolStart(toolName, input) {
    console.log(`Tool ${toolName} started with input:`, input);
  }

  handleToolEnd(toolName, output) {
    console.log(`Tool ${toolName} ended with output:`, output);
  }

  handleLLMEnd(llmName, output) {
    console.log(`LLM ${llmName} ended with output:`, output);
  }

  // Implement other methods as needed
}

const handler = new MyCallbackHandler();

const response = await ragChain.invoke({
  input,
  chat_history: chatHistory.chat_history
}, {
  callbacks: [handler],
});

Example Code with Adjustments

Here is your adjusted code with the tool_choice: "auto" option and a comprehensive callback handler:

const streamingModel = new ChatOpenAI({
  streaming: true,
  modelName: MODEL_NAME,
  temperature: TEMPRATURE,
  verbose: false,
  modelKwargs: {
    seed: 1,
  },
});

const addTool = new DynamicStructuredTool({
  name: "add",
  description: "Adds a and b.",
  schema: z.object({
    a: z.number(),
    b: z.number(),
  }),
  func: async ({ a, b }) => {
    return `${a + b}`;
  },
});

const multiplyTool = new DynamicStructuredTool({
  name: "multiply",
  description: "Can perform multiplications operations.",
  schema: z.object({
    a: z.number(),
    b: z.number(),
  }),
  func: async ({ a, b }) => {
    console.log(`Multiply tool called with input: { a: ${a}, b: ${b} }`);
    const result = a * b * a;
    return result.toString();
  },
});

const tools = [addTool, multiplyTool];

const llmWithTools = streamingModel.bindTools(tools, {
  tool_choice: "auto",
});

const ragChain = await createRetrievalChain({
  retriever: await createHistoryAwareRetriever({
    llm: nonStreamingModel,
    retriever: customRetriever,
    rephrasePrompt: contextualizeQPrompt,
  }),
  combineDocsChain: await createStuffDocumentsChain({
    llm: llmWithTools,
    prompt: qaPrompt,
  }),
});

const chatHistory = await memory.loadMemoryVariables({});

const response = await ragChain.invoke({
  input,
  chat_history: chatHistory.chat_history
}, {
  callbacks: [
    {
      handleLLMNewToken: (token) => {
        console.log(`Received token: ${token}`);
        onTokenStream(token);
      },
      handleLLMEnd: async (output) => {
        console.log('LLM End Output:', JSON.stringify(output, null, 2));
        
        const generation = output.generations[0][0];
        if ('message' in generation) {
          const chatGeneration = generation as ChatGeneration;
          if (chatGeneration.message.additional_kwargs?.tool_calls) {
            for (const toolCall of chatGeneration.message.additional_kwargs.tool_calls) {
              console.log('Tool Call:', JSON.stringify(toolCall, null, 2));
              
              const functionName = toolCall.function.name;
              const args = JSON.parse(toolCall.function.arguments);
              console.log(`Function called: ${functionName}`);
              console.log('Arguments:', args);
              
              // Execute the tool and stream only its result
              const tool = tools.find(t => t.name === functionName);
              if (tool) {
                try {
                  const result = await tool.invoke(args);
                  // Stream only the result
                  onTokenStream(result.toString());
                } catch (error) {
                  console.error(`Error executing tool ${functionName}:`, error);
                }
              } else {
                console.warn(`Tool ${functionName} not found`);
              }
            }
          }
        }
      },
      handleToolStart: (toolStart) => {
        console.log(`Tool Start: ${toolStart.name}`);
        console.log(`Tool Input: ${JSON.stringify(toolStart)}`);
        // Add any additional logging or processing for tool start here
      },
    },
  ],
});

This setup should help ensure that tools are called correctly and that all relevant events are captured during tool invocation ^[1][2][3].

To continue talking to Dosu, mention @dosu.