bug: tool call sometimes not parsed correctly?

[Ricki-BumbleDev]

Issue description

Sometimes the <tool_call>...</tool_call> stuff just gets returned in the response instead of the tool call actually being executed (using Qwen3-14B).

Expected Behavior

Tool call is executed.

Actual Behavior

Tool call is just returned in the response like so:

<tool_call>
{"name": "createTicket", "arguments": {"subject": "...", "description": "...", "assignedTo": "...."}}
</tool_call>

Steps to reproduce

import { defineChatSessionFunction, getLlama, LlamaChatSession, resolveModelFile } from 'node-llama-cpp';
import fs from 'node:fs/promises';

const modelName = 'Qwen3-14B';
const modelPath = await resolveModelFile(`hf:unsloth/${modelName}-GGUF:Q4_K_M`);
const llama = await getLlama();
const model = await llama.loadModel({ modelPath });
const context = await model.createContext();
const systemPrompt = await fs.readFile('./systemPrompt.txt', 'utf8');
const session = new LlamaChatSession({ contextSequence: context.getSequence(), systemPrompt });

const createTicket = defineChatSessionFunction({
  description: 'Creates a new ticket in the project management system',
  params: {
    type: 'object',
    properties: {
      subject: {
        type: 'string',
        description: 'Clear and descriptive subject for the ticket'
      },
      description: {
        type: 'string',
        description: 'Detailed description of what needs to be done'
      },
      assignedTo: {
        enum: ['tech', 'design'],
        description: 'Team to assign the ticket to (tech or design)'
      }
    },
    required: ['subject', 'description', 'assignedTo']
  },
  handler: async params => {
    // Creates the ticket
  }
});

const userInstructions = await fs.readFile('./userInstructions.txt', 'utf8');
const response = await session.prompt(userInstructions, { functions: { createTicket } });
console.log('AI Response:', response);

My Environment

Dependency	Version
Operating System	macOS 24.4.0
CPU	Apple M2 Max
Node.js version	22.13.1
Typescript version	TS not specifically installed, using node's --experimental-strip-types
node-llama-cpp version	3.9.0

npx --yes node-llama-cpp inspect gpu output:

OS: macOS 24.4.0 (arm64)
Node: 22.13.1 (arm64)
node-llama-cpp: 3.9.0

Metal: available

Metal device: Apple M2 Max
Metal used VRAM: 0% (64KB/21.33GB)
Metal free VRAM: 99.99% (21.33GB/21.33GB)
Metal unified memory: 21.33GB (100%)

CPU model: Apple M2 Max
Math cores: 8
Used RAM: 64.54% (20.65GB/32GB)
Free RAM: 35.45% (11.35GB/32GB)
Used swap: 0% (0B/0B)
Max swap size: dynamic
mmap: supported

Additional Context

I believe this has to with when the <tool_call> instruction is right at the start of the response. It looks like for subsequest tool calls it's working.

Relevant Features Used

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, but I don't know how to start. I would need guidance.

[giladgd]

Thanks for reporting this issue!
You were right about it happening only when the function call is at the beginning of the response; I overlooked these kinds of cases in the implementation of the function calling syntax extraction used in the Jinja template chat wrapper.

I've implemented a fix (#472) and will release a new version with it in the next few days.

[github-actions]

🎉 This issue has been resolved in version 3.10.0 🎉

The release is available on:

• v3.10.0
• npm package (@latest dist-tag)
• GitHub release
• v3.10.0

Your semantic-release bot 📦🚀

[louistiti]

Hi @giladgd 👋,

Thanks a lot for all the hard and awesome work you are doing with node-llama-cpp.

I'm using the function calling, but mostly as intent matching and slot management. Meaning that I don't need it to go through the handler and execute the code in the function handler.

When using the version 3.9.0 it works great, as mentioned in this issue, sometimes I just need to parse the <tool_call> tag, but that's okay, at least it provides consistent result.

But when I upgrade it to 3.10.0, then it will sometimes go through the function handler. I understand that since this is basically what it has been made for. I wonder if I should use some JSON mode with structured outputs instead of function calling. Is there anyway to make use of the function calling without executing the code, what would you recommend here? 🙏

Code:

import { getLlama, LlamaChatSession, defineChatSessionFunction } from 'node-llama-cpp'


const modelPath = './models/Qwen_Qwen3-4B-Q4_K_M.gguf'
const jsonSchema = {
    "greet": {
        "description": "Greet the user.",
        handler: async (params) => {
            console.log(`function handler from "greet"`, params)
            return true
        }
    },
    "create_list": {
        "description": "Create a new to-do list based on the given list name.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": {
                    "type": "string",
                    "description": "The name of the to-do list to create."
                }
            }
        },
        handler: async (params) => {
            console.log(`function handler from "create_list"`, params)
            return true
        }
    },
    "get_all_lists": {
        "description": "Retrieve all existing to-do lists.",
        handler: async (params) => {
            console.log(`function handler from "get_all_lists"`, params)
            return true
        }
    },
    "get_list_items": {
        "description": "Retrieve all items from a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": {
                    "type": "string",
                    "description": "The name of the to-do list to retrieve items from."
                }
            }
        },
        handler: async (params) => {
            console.log(`function handler from "get_list_items"`, params)
            return true
        }
    },
    "rename_list": {
        "description": "Rename an existing to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "old_list_name": {
                    "type": "string",
                    "description": "The current name of the to-do list."
                },
                "new_list_name": {
                    "type": "string",
                    "description": "The new name for the to-do list."
                }
            }
        },
        handler: async (params) => {
            console.log(`function handler from "rename_list"`, params)
            return true
        }
    },
    "delete_list": {
        "description": "Delete a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": {
                    "type": "string",
                    "description": "The name of the to-do list to delete."
                }
            }
        },
        handler: async (params) => {
            console.log(`function handler from "delete_list"`, params)
            return true
        }
    },
    "add_todos": {
        "description": "Add items to a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": {
                    "type": "string",
                    "description": "The name of the to-do list to add items to."
                },
                "items": {
                    "type": "array",
                    "items": {
                        "type": "string"
                    },
                    "description": "The items to add to the list."
                }
            }
        },
        handler: async (params) => {
            console.log(`function handler from "add_todos"`, params)
            return true
        }
    },
    "complete_items": {
        "description": "Mark items as completed in a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": {
                    "type": "string",
                    "description": "The name of the to-do list containing the items to complete."
                },
                "items": {
                    "type": "array",
                    "items": {
                        "type": "string"
                    },
                    "description": "The items to mark as completed."
                }
            }
        },
        handler: async (params) => {
            console.log(`function handler from "complete_items"`, params)
            return true
        }
    },
    "uncomplete_items": {
        "description": "Mark items as incomplete in a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": {
                    "type": "string",
                    "description": "The name of the to-do list containing the items to mark as incomplete."
                },
                "items": {
                    "type": "array",
                    "items": {
                        "type": "string"
                    },
                    "description": "The items to mark as incomplete."
                }
            }
        },
        handler: async (params) => {
            console.log(`function handler from "uncomplete_items"`, params)
            return true
        }
    }
}

const llama = await getLlama()
const model = await llama.loadModel({
    modelPath
})
const context = await model.createContext({
    sequences: 1,
    contextSize: 2_048
})
const systemPrompt = `You are a specialized AI assistant that exclusively performs function calling. Your sole purpose is to analyze a user's query and translate it into a structured function call based on a provided list of functions.

You must adhere to the following rules without exception:

1. If parameters are missing, you must return a JSON object indicating which parameters are required:
  \`\`\`json
  {"status": "missing_params", "required_params": ["<param_name_1>", "<param_name_2>"], "name": "<function_name>"}
  \`\`\`
  Replace "<param_name>" with the name of the missing required parameter.
2. If the function is not found, you must return the JSON object:
  \`\`\`json
  {"status": "not_found"}
  \`\`\`
3. You must not invent, assume, create, or infer any value for a parameter that is not explicitly provided by the user.
4. You must only return JSON format. Do not provide any explanations, apologies, greetings, or any other conversational text.

FUNCTIONS:
${jsonSchema}

/no_think`
const session = new LlamaChatSession({
    systemPrompt,
    contextSequence: context.getSequence()
})
const prompt = 'Show me all the lists'
// const prompt = 'Hello there!'
// const prompt = 'Add tomatoes and potatoes to my shopping list'
const res = await session.prompt(prompt, {
    temperature: 0,
    maxTokens: 512,
    functions: jsonSchema
})

console.log('res', res)

Here are the runtime results:

---

When prompt is 'Hello there!'
Result is:

# 3.9.0
{"name": "greet", "arguments": {}}

# 3.10.0
{"name": "greet", "arguments": {}}

When prompt is 'Show me all the lists'
Result is:

# 3.9.0
{"name": "get_all_lists", "arguments": {}}

# 3.10.0
The function is executed, and it shows my console.log "function handler from "get_all_lists" {}"
And the result only contains whitespace

When prompt is 'Add tomatoes and potatoes to my shopping list'
Result is:

# 3.9.0
{"name": "add_todos", "arguments": {"list_name": "my shopping list", "items": ["tomatoes", "potatoes"]}}

# 3.10.0
{"name": "add_todos", "arguments": {"list_name": "my shopping list", "items": ["tomatoes", "potatoes"]}}

---

Another note is that when using 3.10.0, it will sometimes execute the function handler in a loop until it fills the whole context.

[giladgd]

@louistiti When you pass functions to the prompt method, it essentially converts the function definitions into the format the model expects and puts it in the context in the way the model expects (it's not always in the system prompt, depends on the model chat format), and also detects when the model tries to call a function and then forces it to adhere to the strict format and parameters, so it always calls the functions correctly. this is why function calling works so well in node-llama-cpp.

I think the reason you get inconsistent behavior is because you also include additional instructions in the system prompt about how to call functions, so it confuses the model regarding which format to actually use.

The most convenient solution for your use case might be to use .promptWithMeta(...), for each function handler return an empty response ("" or underfined), and then the response in the object you get from .promptWithMeta(...) and find all the function calls.

Here's an example of how you can achieve what you want:

import {
    getLlama, LlamaChatSession, resolveModelFile,
    defineChatSessionFunction, ChatSessionModelFunctions,
    isChatModelResponseFunctionCall
} from "node-llama-cpp";

const modelPath = await resolveModelFile("hf:Qwen/Qwen3-4B-GGUF:Q4_K_M");

const llama = await getLlama();
const model = await llama.loadModel({modelPath});
const context = await model.createContext({
    contextSize: {
        max: 2048
    }
});
const chatSession = new LlamaChatSession({
    systemPrompt:
        "You are a specialized AI assistant that exclusively performs function calling. " +
        "Your sole purpose is to analyze a user's query and call the appropriate functions.",
    contextSequence: context.getSequence()
});

const functions = {
    getAllLists: defineChatSessionFunction({
        description: "Retrieve all existing to-do lists",
        handler: () => void 0
    }),
    getListItem: defineChatSessionFunction({
        description: "Retrieve all items from a specific to-do list",
        params: {
            type: "object",
            properties: {
                listName: {
                    type: "string",
                    description: "The name of the to-do list to retrieve items from"
                }
            }
        },
        handler({listName}) {
            console.log("listName:", listName);
        }
    })
} as const satisfies ChatSessionModelFunctions;

const prompt = "Show me all the lists";
const {response} = await chatSession.promptWithMeta(prompt, {
    temperature: 0,
    maxTokens: 512,
    functions,
    budgets: {
        thoughtTokens: 0
    }
});

const functionCalls = response.filter(isChatModelResponseFunctionCall);
for (const call of functionCalls) {
    console.log("Function call:", call.name);
    console.log("Params:", call.params);
    console.log("Result:", call.result);
}

LlamaChatSession is essentially just a wrapper over LlamaChat that simplifies the API to make the common use cases easy to use. If you want to control the chat inference process in greater control, I recommend you to read the External Chat State guide which covers how to use LlamaChat.
It also covers how to handle function calling yourself; it's important though that you keep the function calls in the chat history, just set an empty response for each.

Regarding the function calling loop, this might be related to the function calling format confusion I mentioned above, but may also be related to the specific model quantization, try using a Q8_0 to see if it helps.

[louistiti]

Hi @giladgd, thanks for your detailed answer. It has been very helpful.

By using LlamaChat, I indeed have more flexibility for my scenario. So I can control what happens before the function calling.

So far the results are what was expected! 👍
Here is my modified code:

import {
    getLlama,
    LlamaChat,
} from 'node-llama-cpp';

const modelPath = './models/Qwen_Qwen3-4B-Q4_K_M.gguf';

const functionSchema = {
    "greet": {
        "description": "Greet the user.",
    },
    "create_list": {
        "description": "Create a new to-do list based on the given list name.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": { "type": "string", "description": "The name of the to-do list to create." }
            },
            "required": ["list_name"]
        }
    },
    "get_all_lists": {
        "description": "Retrieve all existing to-do lists.",
    },
    "get_list_items": {
        "description": "Retrieve all items from a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": { "type": "string", "description": "The name of the to-do list to retrieve items from." }
            },
            "required": ["list_name"]
        }
    },
    "rename_list": {
        "description": "Rename an existing to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "old_list_name": { "type": "string", "description": "The current name of the to-do list." },
                "new_list_name": { "type": "string", "description": "The new name for the to-do list." }
            },
            "required": ["old_list_name", "new_list_name"]
        }
    },
    "delete_list": {
        "description": "Delete a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": { "type": "string", "description": "The name of the to-do list to delete." }
            },
            "required": ["list_name"]
        }
    },
    "add_todos": {
        "description": "Add items to a specific to-do list.",
        "params": {
            "type": "object",
            "properties": {
                "list_name": { "type": "string", "description": "The name of the to-do list to add items to." },
                "items": { "type": "array", "items": { "type": "string" }, "description": "The items to add to the list." }
            },
            "required": ["list_name", "items"]
        }
    },
    "get_weather": {
        "description": "Get the current weather for a specified location.",
        "params": {
            "type": "object",
            "properties": {
                "location": { "type": "string", "description": "The location for which to retrieve the weather." }
            },
            "required": ["location"]
        }
    },
};

const llama = await getLlama();
const model = await llama.loadModel({ modelPath });
const context = await model.createContext({ sequences: 1, contextSize: 2048 });

/*const systemPrompt = "You are a specialized AI assistant that exclusively performs function calling. " +
    "Your sole purpose is to analyze a user's query and call the appropriate functions.";*/
const systemPrompt = `You are a specialized AI assistant that exclusively performs function calling. Your sole purpose is to analyze a user's query and translate it into a structured function call based on a provided list of functions.

You must adhere to the following rules without exception:

1. If parameters are missing, you must return a JSON object indicating which parameters are required:
  \`\`\`json
  {"status": "missing_params", "required_params": ["<param_name_1>", "<param_name_2>"], "name": "<function_name>"}
  \`\`\`
  Replace "<param_name>" with the name of the missing required parameter.
2. If the function is not found, you must return the JSON object:
  \`\`\`json
  {"status": "not_found"}
  \`\`\`
3. You must not invent, assume, create, or infer any value for a parameter that is not explicitly provided by the user.
4. You must only return JSON format. Do not provide any explanations, apologies, greetings, or any other conversational text.`

const llamaChat = new LlamaChat({
    contextSequence: context.getSequence()
});

let chatHistory = llamaChat.chatWrapper.generateInitialChatHistory({
    systemPrompt
});

async function processPrompt(prompt) {
    console.log(`\n--- Processing Prompt: "${prompt}" ---`);

    chatHistory.push({ type: "user", text: prompt });
    chatHistory.push({ type: "model", response: [] });

    const res = await
        llamaChat.generateResponse(chatHistory, {
        temperature: 0.1,
        functions: functionSchema
    });

    // Update the history for potential follow-up questions in the real app
    chatHistory = res.lastEvaluation.cleanHistory;

    // CASE 1: The model decided to call a function.
    if (res.functionCalls && res.functionCalls.length > 0) {
        const call = res.functionCalls[0];
        const functionName = call.functionName;
        const params = call.params;

        console.log(`Model wants to call "${functionName}" with params:`, params);

        const functionDefinition = functionSchema[functionName];

        // This shouldn't happen if the model is well-behaved, but good to check
        if (!functionDefinition) {
            console.log("Final Output (Function not in our schema):");
            console.log(JSON.stringify({ status: "not_found" }, null, 2));
            return;
        }

        const requiredParams = functionDefinition.params?.required || [];
        const missingParams = [];

        for (const required of requiredParams) {
            // Check if the parameter is missing, null, or undefined
            if (params[required] == null) {
                missingParams.push(required);
            }
        }

        // Now, apply our rules
        if (missingParams.length > 0) {
            console.log("Final Output (Missing Parameters):");
            const output = {
                status: "missing_params",
                required_params: missingParams,
                name: functionName
            };
            console.log(JSON.stringify(output, null, 2));
        } else {
            console.log("Final Output (Valid Function Call):");
            const output = {
                status: "success",
                name: functionName,
                params: params
            };
            console.log(JSON.stringify(output, null, 2));
        }

    } else {
        // CASE 2: The model did NOT call a function.
        // It might have followed rule #2 and returned the "not_found" JSON directly.
        console.log("Model did not call a function. Raw text response:", res.response);

        try {
            const parsedResponse = JSON.parse(res.response);
            console.log("Final Output (Parsed from model's text response):");
            console.log(JSON.stringify(parsedResponse, null, 2));
        } catch (e) {
            console.log("Final Output (Model returned non-JSON text):");
            console.log(res.response);
        }
    }
}

await processPrompt("What is the weather like?");
await processPrompt("Add tomatoes and potatoes to my shopping list");
await processPrompt("Show me my lists");
await processPrompt("Tell me a funny story");
await processPrompt('What is in my list?')
await processPrompt('What is in my lesson list?')
await processPrompt('Sorry, actually delete this list')

I'll implement it to my main project and see how it behaves.
I still added a similar system prompt as before. If you have any more recommendations according to my new code, please let me know! 🙏