Feature Request: Make tool calling appended message optional and not default in `llama-server`

[danielhanchen]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

When using --jinja, llama-server appends the following system message if tools are supported: Respond in JSON format, either with tool_call (a request to call tools) or with response reply to the user's request. See

llama.cpp/common/chat.cpp

Line 849 in 12ee176

"Respond in JSON format, either with `tool_call` (a request to call tools) or with `response` reply to the user's request");

Using a custom system message also doesn't work, as the string always gets appended afterwards.

Could this be made optional? Thanks!

Motivation

I couldn't find this in the docs, and I only saw it via --verbose-prompt --verbose and saw it got appended.

Possible Implementation

No response

[pwilkin]

@danielhanchen I'm reworking the template parsing system, so this will get scrapped as a result.

[danielhanchen]

@pwilkin Oh wonderful! :)

[teleprint-me]

Using a custom system message also doesn't work, as the string always gets appended afterwards.

I've been doing some extensive testing via the llama-server REST API and it seems like the system prompt is baked into the jinja template.

If the jinja flag is set, the system prompt is injected from the template into the message sequence. I was able to verify this in the outputs.

{%- if add_generation_prompt -%}
<|start|>assistant
{%- endif -%}, example_format: '<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-12-24

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions

You are a helpful assistant

<|end|><|start|>user<|message|>Hello<|end|><|start|>assistant<|channel|>final<|message|>Hi there<|end|><|start|>user<|message|>How are you?<|end|><|start|>assistant'
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv  update_slots: all slots are idle

I had written a script awhile back that allow introspection of the jinja templates (both rendered and unrendered), but I lost it.

In the webui, if I set the system prompt, it seems like both system prompts end up in the sequence which degrades model performance (the models seemed "confused", so I did a bit of digging).

If I unset the system prompt, and query the model to inform me about the system prompt, it's able to answer my queries regarding those questions.

For example, GPT-OSS will respond as ChatGPT and it will tell me its cutoff data and the current date. It will also identify itself as ChatGPT.

If I add my own system prompt, it's simply appended to the message sequence, identifies itself as ChatGPT, then attempts to adjust itself to the new "system" prompt.

From there, the models performance just nose dives, and the output becomes worse over time.

@pwilkin I looked at the open prs where you're the author and I'm unsure if you're referencing PR #17114.

[pwilkin]

@teleprint-me No, I have not submitted the PR yet

[teleprint-me]

Okay, I appreciate the reply. I actually had a chance to review that PR and it is unrelated. Good to know.

I was able to scavenge my old script from the closed pr and then render the template from the gguf model file.

16:11:03 | /mnt/source/public/python/agent
(.venv) git:(main | Δ) λ python -m agent.hf.render models/gpt-oss-20b-mxfp4.gguf -r
<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-12-24

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions

I am a helpful assistant.

<|end|><|start|>user<|message|>Hello!<|end|><|start|>assistant<|channel|>final<|message|>Hello! How may I assist you today?<|end|><|start|>user<|message|>Can you tell me what pickled mayonnaise is?<|end|><|start|>assistant<|channel|>final<|message|>Certainly! What would you like to know about it?<|end|><|start|>user<|message|>Is it just regular mayonnaise with vinegar or something else?<|end|>

I don't see any pre-emptive tool calls here. I could be missing something, but the base template appends the end user system prompt to the default prompt. I'll have to dig into this a bit more.

For clarity, the jinja template default is as follows:

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-12-24

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions

The user system prompt is then appended to it.

I am a helpful assistant.

This seems like undesirable behavior because a user may not expect this.

I found it surprising because I was previously able to add my own system prompts without obstruction.

If I add my own system prompt with added information, it may conflict with the default prompt. Since this is hard coded, there seem to be very few ways around this.

This should be dynamic instead so users can define their own. If they have not defined a system prompt, then it can default to the base prompt.

Maybe @ngxson can clarify this behavior for us?

[pwilkin]

There is a dedicated Python Qt6 GUI in scripts/jinja/jinja-tester.py for testing templates and their outputs.

[ngxson]

Honestly the existing tool calling system is too strange that this issue is a completely new knowledge to me.

As @pwilkin pointed out, the ongoing rework on tool calling infrastructure may resolve this problem.

Re. how custom system prompt was handled in jinja, I'm not quite sure how it works, but IIRC there was some hack to inject it in the correct place. But ultimately we should probably preserve the same behavior as HF transformers, let the jinja template to handle it.

That to say, with my limited time budget to spend on chat templates, I think I may attempt to rewrite a jinja engine in PEG just to see if it's an improvement or not. Probably make things easier along the way, but not sure.

[teleprint-me]

In the HF templates, Im able to override this behavior, so that might work.

As I understand it (its been awhile since Ive dug into llama.cpp, ggml, or gguf code), the jinja templating system has a custom interface integrated into llama.cpp.

But if I remember correctly, @ochafik wrote the original tool calling and was one of the core collaborators that worked on it along with the minja.hpp interface.

I havent been paying attention because Ive been attempting to learn how the tech works from first principles while working on a few side projects.

Anyways, happy holidays. I appreciate the feedback and work everyones put into this. Its impressive.

[aldehir]

For example, GPT-OSS will respond as ChatGPT and it will tell me its cutoff data and the current date. It will also identify itself as ChatGPT.

This seems like undesirable behavior because a user may not expect this.

This is as designed by OpenAI. They introduced the concept of a developer role that supersedes the system role. The first system message cannot be overwritten in their templates: https://huggingface.co/openai/gpt-oss-20b/blob/main/chat_template.jinja.

I believe this is off-topic. The original post is related to a system message that llama.cpp injects, but in gpt-oss the message is enforced by the model creator.

[teleprint-me]

I wanted to double check, but the model identity is set as a conditional.

And this is not specific to just GPT-OSS. I just used that model as an example.

If you read the template code, which you linked to, the model identity is set as a conditional.

{%- if model_identity is not defined %}
    {%- set model_identity = "You are ChatGPT, a large language model trained by OpenAI." %}
{%- endif %}

Reasoning effort and tool calling are also conditional within the same template.

For example,

{%- if reasoning_effort is not defined %}
    {%- set reasoning_effort = "medium" %}
{%- endif %}

So, they can be overridden. These are just the defaults.

formatted_template = template.render(
    model_identity="I am GPT-OSS, a large language model trained by OpenAI.",
    reasoning_effort="low",
    messages=messages,
    bos_token=bos_token,
    eos_token=eos_token,
    add_generation_prompt=add_generation_prompt,
)

This is just tailored to specifically render GPT's template, but this applies to any template for any model.

21:00:16 | /mnt/source/public/python/agent
(.venv) git:(main | Δ) λ python -m agent.hf.render models/gpt-oss-20b-mxfp4.gguf -r
<|start|>system<|message|>I am GPT-OSS, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-12-25

Reasoning: low

I am a helpful assistant.

The referenced variables matter here.

Its also part of the parent issue as a side comment, which I quoted in my initial comment.

Im also experiencing similar issues which is why I commented in the first place.

Im not as well versed in jinja templates as I used to be (I learned it a long time ago), but Im familiar with it enough having written it by hand myself back before transformers were even a thing.

If I wanted to, I could automate it and bake my own system prompt into the weights Im using, but Id rather understand why I would need to go about doing that in first place before wasting my own time, or any one elses, on the subject.

I fact check the best I can, but Im human and make mistakes. I would prefer to not presume what any vendors intents are — That is off topic.

[aldehir]

If you read the template code, which you linked to, the model identity is set as a conditional.

You're right. If you want to modify specific chat template inputs, you can use --chat-template-kwargs '{"model_identity": ""}' or pass "chat_template_kwargs": {"model_identity": ""} to exclude it in the system prompt.

No model identity via CLI argument

# ran with --chat-template-kwargs '{"model_identity": ""}'
$ curl http://localhost:8080/v1/chat/completions -d '{"model": "gpt-oss-20b", "messages": [{"role":"user","content":"hello"}]}' | jq -r '.__verbose.prompt'
<|start|>system<|message|>
Knowledge cutoff: 2024-06
Current date: 2025-12-24

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>user<|message|>hello<|end|><|start|>assistant

Custom model identity via API request

# passing `"chat_template_kwargs": {"model_identity": "You are a helpful assistant:"}` in the request
$ curl http://localhost:8080/v1/chat/completions -d '{"model": "gpt-oss-20b", "messages": [{"role":"user","content":"hello"}], "chat_template_kwargs": {"model_identity": "You are a helpful assistant"}}' | jq -r '.__verbose.prompt'
<|start|>system<|message|>You are a helpful assistant
Knowledge cutoff: 2024-06
Current date: 2025-12-24

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>user<|message|>hello<|end|><|start|>assistant

No model identity, but a user-defined system prompt via API request

# one more for completeness, no model identity but with a user-defined system message
$ curl http://localhost:8080/v1/chat/completions -d '{"model": "gpt-oss-20b", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role":"user","content":"hello"}], "chat_template_kwargs": {"model_identity": ""}}' | jq -r '.__verbose.prompt'
<|start|>system<|message|>
Knowledge cutoff: 2024-06
Current date: 2025-12-24

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions

You are a helpful assistant

<|end|><|start|>user<|message|>hello<|end|><|start|>assistant

Hope I covered all your use cases! You can do the same for the other variables defined at the top of the template.

I would prefer to not presume what any vendors intents are — That is off topic.

Sorry, I am regurgitating the information here: https://cookbook.openai.com/articles/openai-harmony#system-message-format. No presumptions made, but llama.cpp still exposes a way to modify those values.

And this is not specific to just GPT-OSS.

There-in lies the problem! Each model comes with its own nuances and ultimately the easiest solution is, as @ngxson suggested: let their own Jinja templates deal with it.

If you could provide insight into other models that similarly problematic, that would go a long way in understanding the scope of your problem.

Thank you 😊!

[teleprint-me]

I suspect tool calling is similar in nature to this which is why they experience an injected prompt.

They never specified which model this happens with, but it stands to reason that it is related.

Based on a cursory analysis, its due to the independent variables based on each model, as you noted. This might hint that theres a default in there somewhere.

This is a non-trivial problem because any model can use any variable in their own template. Its solvable with some clever hacking, but might require nuances on a per model basis. It would be nice to automate this, though I have no idea how I might go about solving this problem at the moment.

For example, the reasoning flag in common is very limited, but if we could automate this, then as long as the args match in the model template, it should register assuming the model supports that feature.

Handling these details at runtime wouldnt be easy (for me). Thats why I like asking the authors of the code first. They would best understand their own motivations.

If theres a pr thats supposed to take care of this in the works already, then Im willing to wait. No need to duplicate potential solutions, but if there is no one working on it, then at least its a known limitation.

Also, I appreciate the link to the source material. Ill review it in the mean time. Im still experimenting and exploring llama.cpps latest capabilities.