Can't get tools to work with local model

[samirbajaj]

Hello,

I would like to use a local model (via llama.cpp) as part of the agent I am writing. Unfortunately, I am not having any success.

It is the simplest add/multiply example -- here is the code:

@tool
def add(a: int, b: int) -> int:
    """Adds a and b.

    Args:
        a: first int
        b: second int
    """
    return a + b

@tool
def multiply(a: int, b: int) -> int:
    """Multiplies a and b.

    Args:
        a: first int
        b: second int
    """
    return a * b

@tool
def divide(a: int, b: int) -> float:
    """Divide a and b.

    Args:
        a: first int
        b: second int
    """
    return a / b


tools = [add, multiply, divide]

# Define LLM with bound tools

n_gpu_layers = 1  
n_batch = 512   

# Callbacks support token-wise streaming
callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

llm = ChatLlamaCpp( # <------ LlamaCpp does not have `bind_tools`
    model_path="/path/to/models/phi-4-Q4_K_M.gguf",
    n_gpu_layers=n_gpu_layers,
    n_batch=n_batch,
    f16_kv=True,  # MUST set to True, otherwise you will run into problem after a couple of calls
    callback_manager=callback_manager,
    verbose=True,  # Verbose is required to pass to the callback manager
    temperature=0,
    convert_system_message_to_human=True
)

llm_with_tools = llm.bind_tools(
                    tools
                )             


# System message
sys_msg = SystemMessage(content="You are a helpful assistant tasked with performing arithmetic on a set of inputs.")


def assistant(state: MessagesState):
   return {"messages": [llm_with_tools.invoke([sys_msg] + state["messages"])]}

# Build graph
builder = StateGraph(MessagesState)
builder.add_node("assistant", assistant)
builder.add_node("tools", ToolNode(tools))
builder.add_edge(START, "assistant")
builder.add_conditional_edges(
    "assistant",
    # If the latest message (result) from assistant is a tool call -> tools_condition routes to tools
    # If the latest message (result) from assistant is a not a tool call -> tools_condition routes to END
    tools_condition,
)
builder.add_edge("tools", "assistant")

# Compile graph
react_graph = builder.compile()

messages = [HumanMessage(content="""
    Add 3 and 4. Multiply the output by 2. Divide the output by 5""")]

messages = react_graph.invoke({"messages": messages})

for m in messages['messages']:
    m.pretty_print()

The output is:

Let's go through the steps one by one:

1. **Add 3 and 4:**
   \[
   3 + 4 = 7
   \]

2. **Multiply the output by 2:**
   \[
   7 \times 2 = 14
   \]

3. **Divide the output by 5:**
   \[
   \frac{14}{5} = 2.8
   \]

The final result is \(2.8\).

I tried using create_react_agent, but again don't see any ToolMessages.

I cannot afford to pay for OpenAI models. Please advise.

Thank you.

[magaton]

Could it be that either:

• llm knows the answer (math operations) and decide not to call any of the tools you provided
  or
• phi4 is not actually capable of calling tools?

[SergioRubio01]

Correct, in Ollama it shows that phi4 is not capable of using tools. Try using another model with a tools tag in it

Phi 4:

[SergioRubio01]

But I do not know if this is just for Ollama or it applies to phi4 itself due to its model template

[samirbajaj]

Thank you @SergioRubio01 and @magaton