Tool Calling in GLM-4.5 - Anyone get it to work? #753
Replies: 5 comments 7 replies
-
|
@jordandevai Man, GLM-4.5 is a 110B model, let's face it. Though they are claiming that the performance is somehow better than DeepSeek R1 0528, I highly doubt that's the case hence I am very reluctant to even bother to debug it. Can you please ask this model the following question: "Imagine a runaway trolley is hurtling down a track towards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you pull the lever?". Will it answer correctly? If so, lets debug it. If not, why bother? |
Beta Was this translation helpful? Give feedback.
-
Smaller models can be useful for some tasks, even if their capabilities are limited when compared to larger models. GLM-4.5 Air is a capable model with interesting size - it can fit fully into 96 GB VRAM GPU in pretty usable Q4/Q5 quantitization, so it is a candidate to serve as the fast secondary model for agentic usecases, providing helping hand to a larger model. Well functioning tool calling capabilities are very important to many people, myself included :). |
Beta Was this translation helpful? Give feedback.
-
It is a coding and math model - not really trained on other data as far as I understand, but here's your answer from GLM Air:
curl -X POST http://localhost:8080/completion -H "Content-Type: application/json" -d '{
"prompt": "Answer this question only: Imagine a runaway trolley is hurtling down a track towards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you pull the lever?",
"n_predict": 500,
"temperature": 0.5,
"top_k": 50,
"top_p": 0.9,
"stream": false
}' | jq '.'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4604 100 4244 100 360 401 34 0:00:10 0:00:10 --:--:-- 974
{
"index": 0,
"content": " Why or why not?\nOkay, the user is asking about the classic trolley problem scenario. They want a direct answer to whether I'd pull the lever and why. \n\nHmm, this is a famous ethical dilemma that pits utilitarianism against deontology. The user seems to be testing my moral reasoning - maybe they're studying philosophy or just curious about AI ethics. \n\nLet me break it down: Pulling the lever saves five lives but kills one. Not pulling it lets five die. The core conflict is between maximizing overall good (utilitarian view) and not actively causing harm (deontological view). \n\nI notice the user specified \"dead people\" vs \"living person\" - that's interesting. They're emphasizing the moral weight of killing a living person versus letting dead people die. Smart framing. \n\nFor my response, I should:\n- Give a clear yes/no first (pull the lever)\n- Explain the utilitarian reasoning simply\n- Acknowledge the moral complexity\n- Avoid sounding dismissive of the deontological view\n\nThe user might be looking for more than just an answer - perhaps they want to understand how different ethical frameworks work. But since they asked for only the answer, I'll keep it focused. \n\nI wonder if they're preparing for a debate or writing a paper. The question feels academic. Maybe they'd appreciate knowing this is a real philosophical thought experiment used in ethics classes. \n\nBetter avoid getting too technical though. The key is showing I recognize the dilemma's gravity while giving a reasoned stance. The \"why\" matters as much as the \"what\" here. \n\n...time to structure the response. Clear position first, then concise reasoning that honors both sides of the dilemma. No fluff.</think>**Yes, I would pull the lever.** \n\n### Why? \n1. **Utilitarian Perspective**: The action maximizes overall good. Saving five lives at the cost of one results in a net gain of four lives. Morally, this minimizes harm and maximizes well-being. \n2. **Active vs. Passive Harm**: While pulling the lever directly causes one death, *not* acting allows five deaths to occur passively. Philosophically, active harm is often weighed differently than passive inaction, but the core ethical question is about preventing greater loss. \n3. **Intent and Responsibility**: The goal is to save lives, not to kill. The death of the one person is a tragic but unintended consequence of redirecting",
"tokens": [],
"id_slot": 0,
"stop": true,
"model": "gpt-3.5-turbo",
"tokens_predicted": 500,
"tokens_evaluated": 51,
"generation_settings": {
"n_predict": 500,
"seed": 4294967295,
"temperature": 0.5,
"dynatemp_range": 0.0,
"dynatemp_exponent": 1.0,
"top_k": 50,
"top_p": 0.8999999761581421,
"min_p": 0.05000000074505806,
"top_n_sigma": -1.0,
"xtc_probability": 0.0,
"xtc_threshold": 0.10000000149011612,
"typical_p": 1.0,
"repeat_last_n": 64,
"repeat_penalty": 1.0,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"dry_multiplier": 0.0,
"dry_base": 1.75,
"dry_allowed_length": 2,
"dry_penalty_last_n": 64000,
"dry_sequence_breakers": [
"\n",
":",
"\"",
"*"
],
"mirostat": 0,
"mirostat_tau": 5.0,
"mirostat_eta": 0.10000000149011612,
"stop": [],
"max_tokens": 500,
"n_keep": 0,
"n_discard": 0,
"ignore_eos": false,
"stream": false,
"logit_bias": [],
"n_probs": 0,
"min_keep": 0,
"grammar": "",
"grammar_lazy": false,
"grammar_triggers": [],
"preserved_tokens": [],
"chat_format": "Content-only",
"reasoning_format": "auto",
"reasoning_in_content": false,
"thinking_forced_open": false,
"samplers": [
"penalties",
"dry",
"top_n_sigma",
"top_k",
"typ_p",
"top_p",
"min_p",
"xtc",
"temperature"
],
"speculative.n_max": 16,
"speculative.n_min": 0,
"speculative.p_min": 0.75,
"timings_per_token": false,
"post_sampling_probs": false,
"lora": []
},
"prompt": "Answer this question only: Imagine a runaway trolley is hurtling down a track towards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you pull the lever?",
"has_new_line": true,
"truncated": false,
"stop_type": "limit",
"stopping_word": "",
"tokens_cached": 550,
"timings": {
"prompt_n": 51,
"prompt_ms": 984.105,
"prompt_per_token_ms": 19.296176470588236,
"prompt_per_second": 51.82373832060603,
"predicted_n": 500,
"predicted_ms": 9576.744,
"predicted_per_token_ms": 19.153488000000003,
"predicted_per_second": 52.20981160193903
}
}
|
Beta Was this translation helpful? Give feedback.
-
The word "capable", I believe, can be safely omitted.
AFAIK, the math and coding bumps the performance of the model in other areas, but not the other way around. |
Beta Was this translation helpful? Give feedback.
-
I understand. I do not use LLMs for coding either. Tool calling is a useful feature in various other areas though. |
Beta Was this translation helpful? Give feedback.
-
|
Are you saying Deepseek R1 performs better? A lot of coders prefer GLM from what I've seen (anecdotally of course). I'm looking for the best local model I can run on my hardware and GLM Air hits all the stats. |
Beta Was this translation helpful? Give feedback.
-
Yes, this is exactly what I am saying.
I don't watch youtube. I don't trust the benchmarks. I use bash and C, I don't do too much frontend. So in my personal experience, DeepSeek R1/V3/V3.1 performs much better. |
Beta Was this translation helpful? Give feedback.
-
|
I don't watch or trust benchmarks, either. I do a lot of app development in both frontend and backend work - e.g.: Python, Clarity, HTTP Requests, OS Scripting, React, Data analysis, etc. I don't really touch low level or mid level languages anymore. Just looking for the most efficient LLM for my home coding dev station is all. I figured a 105B Parameter LLM quantized to Q4 with Q5/Q8 cache would hit the marks! |
Beta Was this translation helpful? Give feedback.
-
Okay cool, I heard you. Probably you're right, sometimes the speed is required. Okay I will download some quants and give it a test in the near future. |
Beta Was this translation helpful? Give feedback.
-
|
ggml-org/llama.cpp#15186 hasn't been merged in |
Beta Was this translation helpful? Give feedback.
-
|
It looks like the grammar trigger pattern is wrong (testing GLM-4.6). The actual response from LLM with a tool call is: The LLM doesn't respond with [ |
Beta Was this translation helpful? Give feedback.
-
I've been running GLM 4.5 Air (106B A12B?) on a patched llama.cpp, and tool calling works fine. There are very few good coding models in that size range, but GLM 4.5 Air works as well as anything I've tried. I set it up about 2 months ago, so the details have probably changed now, but here's what I did:
template.jinja (old, newer versions exist)[gMASK]<sop>
{%- if tools -%}
<|system|>
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{% for tool in tools %}
{{ tool | tojson }}
{% endfor %}
</tools>
For each function call, output the function name and arguments within the following XML format:
<tool_call>{function-name}
<arg_key>{arg-key-1}</arg_key>
<arg_value>{arg-value-1}</arg_value>
<arg_key>{arg-key-2}</arg_key>
<arg_value>{arg-value-2}</arg_value>
...
</tool_call>{%- endif -%}
{%- macro visible_text(content) -%}
{%- if content is string -%}
{{- content }}
{%- elif content is iterable and content is not mapping -%}
{%- for item in content -%}
{%- if item is mapping and item.type == 'text' -%}
{{- item.text }}
{%- elif item is string -%}
{{- item }}
{%- endif -%}
{%- endfor -%}
{%- else -%}
{{- content }}
{%- endif -%}
{%- endmacro -%}
{%- set ns = namespace(last_user_index=-1) %}
{%- for m in messages %}
{%- if m.role == 'user' %}
{%- set user_content = visible_text(m.content) -%}
{%- if not ("tool_response" in user_content) %}
{% set ns.last_user_index = loop.index0 -%}
{%- endif -%}
{%- endif %}
{%- endfor %}
{% for m in messages %}
{%- if m.role == 'user' -%}<|user|>
{%- set user_content = visible_text(m.content) -%}
{{ user_content }}
{%- if enable_thinking is defined and not enable_thinking -%}
{%- if not user_content.endswith("/nothink") -%}
{{- '/nothink' -}}
{%- endif -%}
{%- endif -%}
{%- elif m.role == 'assistant' -%}
<|assistant|>
{%- set reasoning_content = '' %}
{%- set content = visible_text(m.content) %}
{%- if m.reasoning_content is string %}
{%- set reasoning_content = m.reasoning_content %}
{%- else %}
{%- if '</think>' in content %}
{%- set think_parts = content.split('</think>') %}
{%- if think_parts|length > 1 %}
{%- set before_end_think = think_parts[0] %}
{%- set after_end_think = think_parts[1] %}
{%- set think_start_parts = before_end_think.split('<think>') %}
{%- if think_start_parts|length > 1 %}
{%- set reasoning_content = think_start_parts[-1].lstrip('\n') %}
{%- endif %}
{%- set content = after_end_think.lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- endif %}
{%- if loop.index0 > ns.last_user_index and reasoning_content -%}
{{ '\n<think>' + reasoning_content.strip() + '</think>'}}
{%- else -%}
{{ '\n<think></think>' }}
{%- endif -%}
{%- if content.strip() -%}
{{ '\n' + content.strip() }}
{%- endif -%}
{% if m.tool_calls %}
{% for tc in m.tool_calls %}
{%- if tc.function %}
{%- set tc = tc.function %}
{%- endif %}
{{ '\n<tool_call>' + tc.name }}
{% set _args = tc.arguments %}
{% for k, v in _args.items() %}
<arg_key>{{ k }}</arg_key>
<arg_value>{{ v | tojson if v is not string else v }}</arg_value>
{% endfor %}
</tool_call>{% endfor %}
{% endif %}
{%- elif m.role == 'tool' -%}
{%- if m.content is string -%}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|observation|>' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- m.content }}
{{- '\n</tool_response>' }}
{%- else -%}
<|observation|>{% for tr in m.content %}
<tool_response>
{{ tr.output if tr.output is defined else tr }}
</tool_response>{% endfor -%}
{% endif -%}
{%- elif m.role == 'system' -%}
<|system|>
{{ visible_text(m.content) }}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
<|assistant|>{{- '\n<think></think>' if (enable_thinking is defined and not enable_thinking) else '' -}}
{%- endif -%}llama-server arguments (probably not the fastest)ResultsThis is the strongest local coding model I've gotten running with a combination of a 3090 and a Ryzen9 9900X with 64GB of RAM. Tool calling is nearly 100% reliable using Cline. In general, it's a better coder than Qwen3 30B A3B and GPT OSS 20B. Obviously, it's not Sonnet 4.5: It misses the point just often enough to be annoying, and I run out of context window before it can finish a complete feature end-to-end. But it's a useful model, even if it pushes my system to the absolute limit. My main use case is "first drafts of code that would be tedious to implement by hand". I assume that I'll be reading all the code closely and refactoring it a bit, but I'd do that anyway for production code. Mostly I ask it to write Rust and typed Python, plus an occasional bit of TypeScript. Obviously, my system won't run a full-sized DeepSeek. I've heard rumors of people getting a really tiny quant of a full-sized GLM 4.5 (355B A32B) running with a 3090 and ton of system RAM, but I've never tried. I also want to take a look at GPT 120 OSS at some point, which is the natural competitor for GLM 4.5 Air. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hey all - love the project and the speed of IK_LLama over Llama.cpp! However, spent the weekend trying to get tool calling to work to no avail with GLM-4.5-Air.
It does things like this:
And then breaks Continue.dev. I'm running the latest build of IK_Llama:
Anyone else overcome this? Do I need to build a proxy? I reviewed several PRs that said they fixed the GLM Tool calls. Here is my continue dev config if it helps:
Beta Was this translation helpful? Give feedback.
All reactions