🛠️ Discussion on Tool support

[anakin87]

UPDATE: All work on Tool has been incorporated into Haystack 2.9.0. See the docs

---

This is the discussion board for a group of experimental components and dataclasses to improve the support for Tool calling in Haystack.

What is Tool calling/function calling?
A growing number of Language Models now support passing tool definitions alongside the prompt.
Tools can be functions or APIs.
When appropriate, the model can generate a call to the tool in order to address the user’s query, although the model itself does not execute the tool.

The features outlined below are built to support tools in Haystack in a unified way, covering the entire process from tool call preparation to proper invocation and potential post-processing.

Experimental features:

• refactored ChatMessage dataclass to be more flexible and future-proof.
• Tool dataclass: a simple and unified abstraction to represent Tools throughout the framework.
• refactored OpenAIChatGenerator, with support for tools.
• ToolInvoker component, to actually execute tool calls prepared by Language Models.

🚧 We are working on adding support for more Chat Generators and developing easier ways to convert functions into tools...

🧑‍🍳 Try the Cookbook here. Contains a step by step guide to the new features and shows several end-to-end examples.

🧑‍🍳 New recipe: Newsletter Sending Agent

Let us know how you get along with these components, and please do let us know if you have any feedback.

Note: Experimental features live in this repository for a fixed period of time. We don't guarantee that we will continue maintaining experimental features. But if they are successful, stable and if you like it, we will move the feature to the core Haystack package.

[cjayb]

Very much appreciate this abstraction! Makes working with, and especially debugging, tool calls a lot more transparent. I need to use Azure OpenAI, but managed to port #57 over (see my comment to #57 if you'd like me to contribute it back?)

[anakin87]

@cjayb Thanks for the appreciation!

Currently, this feature is experimental and we want to support it in 3 Chat Generators: OpenAI, Antrophic, and Ollama.
We will use the experiment time to iterate quickly and get feedback.

If the experiment is successful, starting in November, we will implement it for other Generators as well and I can ping you to contribute.

[cjayb]

Sounds like a plan 👏

[anakin87]

During the live, a community member mentioned that he would like to use the @tool decorator in addition to the class method Tool.from_function

[anakin87]

🧑‍🍳 New recipe: Newsletter Sending Agent with Experimental Haystack Tools

[anakin87]

We also added support for Tools in Ollama, Anthropic and Hugging Face API Chat Generators!

[Emil-io]

Idea: Output of Tool Invoker matches input of ChatGenerator

The proper workflow when using tools requires 2 generations:
(1) The LLM outputs that a function call should be performed
(2) Based on the chat history + Last LLM reply + Tool Result a new generation is started, which outputs a response text message

For this to properly work, I have two options
(a) Use two separate Generator components in my pipeline. First generator component is for Step (1), after that there is a router, which routes the output to Generator 2 for Step (2) , if in Step (1) a tool call was requested (Not preferred)
(b) Use the same generator with a loop. Easiest would be, if the function call output of Generator (1) could simply be passed to the Tool Invoker. And the the output of the tool invoker can then directly be provided to Generator (2) for the generation of text

Since (b) is preferred:
This would mean, that the Tool Invoker outputs the Chat History + Last LLM reply + Tool Result.
I think this would make the workflow for working with tools very smooth (only one routing after Generator (1) and then a very simple loop by directly connecting the Tool Invoker to the Generator again)

Alternatively, a new component called OpenAIAgent, which for Step (2) automatically processes the output of the tool invoker in its current type, and then also turns it into the appropriate chat message list based on the chat messages it was provided with for first generation

P.S. Maybe routing could even be performed by the Generator, since this follows a fixed syntax (either for ChatGeneraotr or OpenAIAgent :) )

[Emil-io]

@anakin87 wdyt?

[anakin87]

I will have a proper look this week

[farzbood]

Hi,
I've tried the cookbook code with a number of HuggingFace open models (HuggingFaceH4/starchat2-15b-v0.1 , mistralai/Mistral-7B-Instruct-v0.3, Qwen/QwQ-32B-Preview) , it doesn't work as expected with any of them. (the conversation stuck with joker with a response says the request can't be processed and back to the User prompt, in the best case scenario, dialog transferred to Refund Agent but still says it can't process the refund request)
Is this type of collaborative agent model depends on a specific LLM capability or provider?

[anakin87]

UPDATE: All work on Tool has been incorporated into Haystack 2.9.0. See the docs.