Eval bug: Tools In Prompt Crashing On gpt-oss 20b

[kennethgoodman]

Name and Version

Ran with:

$ git pull
$ ./build/bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 0 -fa --jinja --reasoning-format none -v --verbose-prompt > tmp.log 2>&1

This happens when I use Claude-code to route requests so there is a lot of tokens being sent here

Llama.cpp verbose logs:

(base) kennethgoodman@kenneths-mbp-4 llama.cpp % tail -f tmp.log
slot process_toke: id  0 | task 652 | n_decoded = 54, n_remaining = 31946, next token: 200002 ''
slot      release: id  0 | task 652 | stop processing: n_past = 11197, truncated = 0
slot print_timing: id  0 | task 652 | 
prompt eval time =   13836.82 ms / 11063 tokens (    1.25 ms per token,   799.53 tokens per second)
       eval time =     823.03 ms /    54 tokens (   15.24 ms per token,    65.61 tokens per second)
      total time =   14659.85 ms / 11117 tokens
srv  update_chat_: Parsing chat message: <|channel|>analysis<|message|>The user says "hi". It's a greeting. We should respond simply, probably "Hello!" or similar. According to guidelines, greeting response: maybe just "Hello!" We should not mention todo or anything.<|start|>assistant<|channel|>final<|message|>Hello!
Parsing input with format GPT-OSS: <|channel|>analysis<|message|>The user says "hi". It's a greeting. We should respond simply, probably "Hello!" or similar. According to guidelines, greeting response: maybe just "Hello!" We should not mention todo or anything.<|start|>assistant<|channel|>final<|message|>Hello!
(lldb) process attach --pid 5320
libc++abi: terminating due to uncaught exception of type std::runtime_error: Unexpected content at end of input

Claude code logs:

[2025-08-05T21:37:04.228Z] Original Response: {
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": null
      }
    }
  ],
  "created": 1754429824,
  "id": "chatcmpl-nRGxMwrj1j6M96P70YtFvR3TmCRbVNJJ",
  "model": "gpt-oss-20b",
  "system_fingerprint": "b6096-fd1234cb",
  "object": "chat.completion.chunk"
}
[2025-08-05T21:37:04.229Z] send data: event: message_start
data: {"type":"message_start","message":{"id":"msg_1754429809537","type":"message","role":"assistant","content":[],"model":"gpt-oss-20b","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":1,"output_tokens":1}}}
[2025-08-05T21:37:05.802Z] cancle stream: null

Operating systems

Mac

GGML backends

Metal

Hardware

Apple M2 Max 96GB

Models

ggml-org/gpt-oss-20b-GGUF

Problem description & steps to reproduce

Build:

$ git pull
$ cmake --build build --config Release -j 8
$ ./build/bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 0 -fa --jinja --reasoning-format none -v --verbose-prompt > tmp.log 2>&1

claude code router config:

{
  "LOG": true,
  "API_TIMEOUT_MS": 600000,
  "Providers": [
    {
      "name": "ollama",
      "api_base_url": "http://127.0.0.1:8080/v1/chat/completions",
      "api_key": "ollama",
      "models": ["gpt-oss-20b"],
    }
  ],
  "Router": {
    "default": "ollama,gpt-oss-20b",
  }
}

running claude code router:

$ ccr code

Then just any message like "hi" will crash

First Bad Commit

No response

Relevant log output

prompt eval time =   13836.82 ms / 11063 tokens (    1.25 ms per token,   799.53 tokens per second)
       eval time =     823.03 ms /    54 tokens (   15.24 ms per token,    65.61 tokens per second)
      total time =   14659.85 ms / 11117 tokens
srv  update_chat_: Parsing chat message: <|channel|>analysis<|message|>The user says "hi". It's a greeting. We should respond simply, probably "Hello!" or similar. According to guidelines, greeting response: maybe just "Hello!" We should not mention todo or anything.<|start|>assistant<|channel|>final<|message|>Hello!
Parsing input with format GPT-OSS: <|channel|>analysis<|message|>The user says "hi". It's a greeting. We should respond simply, probably "Hello!" or similar. According to guidelines, greeting response: maybe just "Hello!" We should not mention todo or anything.<|start|>assistant<|channel|>final<|message|>Hello!
(lldb) process attach --pid 5320
libc++abi: terminating due to uncaught exception of type std::runtime_error: Unexpected content at end of input

[x0wllaar]

Try removing --jinja (it won't work well / will produce gibberish, but just to see if it crashes or not)

[kennethgoodman]

I get

srv  log_server_r: response: {"error":{"code":500,"message":"tools param requires --jinja flag","type":"server_error"}}

No crash

Btw, with --jinja this will crash my entire terminal application not the specific tab

[marceldev89]

Same thing on Linux btw.

[x0wllaar]

tools param requires --jinja flag

This is probably specific to the server, with llama-cli (on Windows) (./llama-cli --model /path/to/gguf/01.gguf -ngl 999 --n-cpu-moe 32 --ctx-size 131072 -fa --jinja --reasoning-format none) I get a response, then a crash before stats are shown, same command without --jinja --reasoning-format none does not crash

[kennethgoodman]

Crashes on this payload:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "learn about the reddit api"
        }
      ]
    }
  ],
  "model": "gpt-oss-20b",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "mcp__redditgithub__fetch_generic_url_content",
        "description": "Generic tool to fetch content from any absolute URL",
        "parameters": {
          "type": "object",
          "properties": {
            "url": {
              "type": "string",
              "description": "The URL of the document or page to fetch"
            }
          },
          "required": [
            "url"
          ],
          "additionalProperties": false,
          "$schema": "http://json-schema.org/draft-07/schema#"
        }
      }
    }
  ]
}

logs:

Parsing input with format GPT-OSS: <|channel|>analysis<|message|>User says "hi". They likely want a response. We can respond politely.<|start|>assistant<|channel|>final<|message|>Hello! How can I help you today?
(lldb) process attach --pid 15735

curl -X POST http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d @data.json

[kennethgoodman]

Does not fail on:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "hi"
        }
      ]
    }
  ],
  "tools": []
}

response:

srv  update_chat_: Parsing chat message: <|channel|>analysis<|message|>User says "hi". They likely want a response. We can respond politely.<|start|>assistant<|channel|>final<|message|>Hello! How can I help you today?

So it seems that tool calls are what's broken

[kennethgoodman]

interestingly:

if I run:

./build/bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 0 -fa --jinja -v --verbose-prompt > tmp.log 2>&1

NO --reasoning-format none

And send:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "call the function get_weather to get weather"
        }
      ]
    }
  ],
  "model": "gpt-oss-20b",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Generic tool to fetch weather",
        "parameters": {
          "type": "object",
          "properties": {},
          "required": [],
          "additionalProperties": false,
          "$schema": "http://json-schema.org/draft-07/schema#"
        }
      }
    }
  ]
}

this works!

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": "The user says: \"call the function get_weather to get weather\". The function signature seems incomplete: \"type get_weather = (_: { , }) => any;\". That seems odd: maybe they omitted arguments? Possibly no arguments needed. We can call get_weather with empty object. Let's call.<|start|>assistant<|channel|>commentary to=functions.get_weather json<|message|>{\"\":\"\"}",
        "content": ""
      }
    }
  ],
  "created": 1754449916,
  "model": "gpt-oss-20b",
  "system_fingerprint": "b6096-fd1234cb",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 83,
    "prompt_tokens": 125,
    "total_tokens": 208
  },
  "id": "chatcmpl-taMCiuppmiswKIVILLkM8DFIw3ggh7Lc",
  "__verbose": {
    "index": 0,
    "content": "<|channel|>analysis<|message|>The user says: \"call the function get_weather to get weather\". The function signature seems incomplete: \"type get_weather = (_: { , }) => any;\". That seems odd: maybe they omitted arguments? Possibly no arguments needed. We can call get_weather with empty object. Let's call.<|start|>assistant<|channel|>commentary to=functions.get_weather json<|message|>{\"\":\"\"}",
    "tokens": [],
    "id_slot": 0,
    "stop": true,
    "model": "gpt-oss-20b",
    "tokens_predicted": 83,
    "tokens_evaluated": 125,
    "generation_settings": {
      "n_predict": -1,
      "seed": 4294967295,
      "temperature": 0.800000011920929,
      "dynatemp_range": 0,
      "dynatemp_exponent": 1,
      "top_k": 40,
      "top_p": 0.949999988079071,
      "min_p": 0.05000000074505806,
      "top_n_sigma": -1,
      "xtc_probability": 0,
      "xtc_threshold": 0.10000000149011612,
      "typical_p": 1,
      "repeat_last_n": 64,
      "repeat_penalty": 1,
      "presence_penalty": 0,
      "frequency_penalty": 0,
      "dry_multiplier": 0,
      "dry_base": 1.75,
      "dry_allowed_length": 2,
      "dry_penalty_last_n": 131072,
      "dry_sequence_breakers": [
        "\n",
        ":",
        "\"",
        "*"
      ],
      "mirostat": 0,
      "mirostat_tau": 5,
      "mirostat_eta": 0.10000000149011612,
      "stop": [],
      "max_tokens": -1,
      "n_keep": 0,
      "n_discard": 0,
      "ignore_eos": false,
      "stream": false,
      "logit_bias": [],
      "n_probs": 0,
      "min_keep": 0,
      "grammar": "",
      "grammar_lazy": false,
      "grammar_triggers": [],
      "preserved_tokens": [],
      "chat_format": "GPT-OSS",
      "reasoning_format": "auto",
      "reasoning_in_content": false,
      "thinking_forced_open": false,
      "samplers": [
        "penalties",
        "dry",
        "top_n_sigma",
        "top_k",
        "typ_p",
        "top_p",
        "min_p",
        "xtc",
        "temperature"
      ],
      "speculative.n_max": 16,
      "speculative.n_min": 0,
      "speculative.p_min": 0.75,
      "timings_per_token": false,
      "post_sampling_probs": false,
      "lora": []
    },
    "prompt": "<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.\nKnowledge cutoff: 2024-06\nCurrent date: 2025-08-05\n\nreasoning: medium\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.\nCalls to these tools must go to the commentary channel: 'functions'.<|end|><|start|>developer<|message|>\n\n# Tools\n\n## functions\n\nnamespace functions {\n\n// Generic tool to fetch weather\ntype get_weather = (_: {\n,\n}) => any;\n\n} // namespace functions<|end|><|start|>user<|message|>call the function get_weather to get weather<|end|><|start|>assistant",
    "has_new_line": false,
    "truncated": false,
    "stop_type": "eos",
    "stopping_word": "",
    "tokens_cached": 207,
    "timings": {
      "prompt_n": 1,
      "prompt_ms": 12.968,
      "prompt_per_token_ms": 12.968,
      "prompt_per_second": 77.11289327575571,
      "predicted_n": 83,
      "predicted_ms": 1023.838,
      "predicted_per_token_ms": 12.335397590361445,
      "predicted_per_second": 81.06751263383465
    }
  },
  "timings": {
    "prompt_n": 1,
    "prompt_ms": 12.968,
    "prompt_per_token_ms": 12.968,
    "prompt_per_second": 77.11289327575571,
    "predicted_n": 83,
    "predicted_ms": 1023.838,
    "predicted_per_token_ms": 12.335397590361445,
    "predicted_per_second": 81.06751263383465
  }
}

BUT:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "hi"
        }
      ]
    }
  ],
  "model": "gpt-oss-20b",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "mcp__redditgithub__fetch_generic_url_content",
        "description": "Generic tool to fetch content from any absolute URL.",
        "parameters": {
          "type": "object",
          "properties": {},
          "required": [],
          "additionalProperties": false,
          "$schema": "http://json-schema.org/draft-07/schema#"
        }
      }
    }
  ]
}

fails:

request: {  "messages": [    {      "role": "user",      "content": [        {          "type": "text",          "text": "hi"        }      ]    }  ],  "model": "gpt-oss-20b",  "tools": [    {      "type": "function",      "function": {        "name": "mcp__redditgithub__fetch_generic_url_content",        "description": "Generic tool to fetch content from any absolute URL.",        "parameters": {          "type": "object",          "properties": {},          "required": [],          "additionalProperties": false,          "$schema": "http://json-schema.org/draft-07/schema#"        }      }    }  ]}
srv  params_from_: Grammar:
srv  params_from_: Grammar lazy: false
srv  params_from_: Chat format: GPT-OSS
srv  add_waiting_: add task 0 to waiting list. current waiting = 0 (before add)
que          post: new task, id = 0/1, front = 0
que    start_loop: processing new tasks
que    start_loop: processing task, id = 0
slot get_availabl: id  0 | task -1 | selected slot by lru, t_last = -1
slot        reset: id  0 | task -1 |
slot launch_slot_: id  0 | task 0 | launching slot : {"id":0,"id_task":0,"n_ctx":131072,"speculative":false,"is_processing":false,"params":{"n_predict":-1,"seed":4294967295,"temperature":0.800000011920929,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":131072,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"stop":[],"max_tokens":-1,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"logit_bias":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"chat_format":"GPT-OSS","reasoning_format":"auto","reasoning_in_content":false,"thinking_forced_open":false,"samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.n_max":16,"speculative.n_min":0,"speculative.p_min":0.75,"timings_per_token":false,"post_sampling_probs":false,"lora":[]},"prompt":"<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.\nKnowledge cutoff: 2024-06\nCurrent date: 2025-08-05\n\nreasoning: medium\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.\nCalls to these tools must go to the commentary channel: 'functions'.<|end|><|start|>developer<|message|>\n\n# Tools\n\n## functions\n\nnamespace functions {\n\n// Generic tool to fetch content from any absolute URL.\ntype mcp__redditgithub__fetch_generic_url_content = (_: {\n,\n}) => any;\n\n} // namespace functions<|end|><|start|>user<|message|>hi<|end|><|start|>assistant","next_token":{"has_next_token":true,"has_new_line":false,"n_remain":-1,"n_decoded":0,"stopping_word":""}}
slot launch_slot_: id  0 | task 0 | processing task

...

srv  update_chat_: Parsing chat message: <|channel|>analysis<|message|>User says "hi". We need a friendly greeting.<|start|>assistant<|channel|>final<|message|>Hello! 👋 How can I help you today?
Parsing input with format GPT-OSS: <|channel|>analysis<|message|>User says "hi". We need a friendly greeting.<|start|>assistant<|channel|>final<|message|>Hello! 👋 How can I help you today?
(lldb) process attach --pid 25579
libc++abi: terminating due to uncaught exception of type std::runtime_error: Unexpected content at end of input

[kennethgoodman]

I think what I've gathered is that this fails if:

• there are tool calls in request AND no tool calls in response
• there are tool calls in request AND --reasoning-format none is true

[hmmroger]

It's indeed to do with tool calls, not so much with the --reasoning-format since it's just a side effect. The issue is in common_chat_parse_gpt_oss(common_chat_msg_parser & builder) where tool calls parsing is missing. I think for gpt-oss, we need channel state while parsing and account for multiple set of channels in output. I took a quick look, but it seems a bit more involved to get everything working with streaming mode in llama-server so my recommendation is to use --reasoning-format none and add the following in the function for immediate fix. Then perform parsing in the client.

builder.add_content(builder.consume_rest());

[victorb]

I hit the same issue, and from the comment above, I did this (which I have no idea if it's correct or not, I know nothing about llama.cpp internals) which seems to have made it stop crashing at least.

diff --git a/common/chat.cpp b/common/chat.cpp
index 316bd241..ce34484b 100644
--- a/common/chat.cpp
+++ b/common/chat.cpp
@@ -1321,6 +1321,7 @@ static common_chat_params common_chat_params_init_gpt_oss(const common_chat_temp
 static void common_chat_parse_gpt_oss(common_chat_msg_parser & builder) {
     // TODO @ngxson : this won't work with --special enabled, we should fix that
     builder.try_parse_reasoning("<|channel|>analysis<|message|>", "<|start|>assistant<|channel|>final<|message|>");
+    builder.add_content(builder.consume_rest());
     if (!builder.syntax().parse_tool_calls) {
         builder.add_content(builder.consume_rest());
         return;

Then running gpt-oss-120b like this (on revision 1d72c841888b9450916bdd5a9b3274da380f5b36):

CUDA_VISIBLE_DEVICES="0" ./build/bin/llama-server -fa --gpu-layers 99 --threads $(nproc) --threads-batch $(nproc) --batch-size 4096 --ctx-size 131072 --jinja -v --verbose-prompt --model /mnt/nas/models/lmstudio-community/gpt-oss-120b-GGUF/gpt-oss-120b-MXFP4-00001-of-00002.gguf

Seems tool-calling works correctly without crashing then (but without the patch above, it crashes).

Example

ChatCompletionRequest {
    model: "gpt-oss-20b-MXFP4.gguf",
    max_tokens: 4096,
    messages: [
        ChatMessage {
            role: "system",
            content: "You are a helpful assistant. Reasoning: high\n\n# Valid channels: analysis, commentary, final.\nCalls to these tools must go to the commentary channel: 'functions'.",
            channel: None,
            recipient: None,
            tool_calls: None,
            tool_call_id: None,
        },
        ChatMessage {
            role: "developer",
            content: "# Instructions\nAnswer calmly and use tools when helpful.",
            channel: None,
            recipient: None,
            tool_calls: None,
            tool_call_id: None,
        },
        ChatMessage {
            role: "user",
            content: "What is 5123 + 12356?",
            channel: None,
            recipient: None,
            tool_calls: None,
            tool_call_id: None,
        },
        ChatMessage {
            role: "assistant",
            content: "",
            channel: Some(
                "commentary",
            ),
            recipient: None,
            tool_calls: Some(
                [
                    ToolCall {
                        id: "call_0",
                        type: "function",
                        function: ToolCallFunction {
                            name: "add",
                            arguments: "{\"a\":5123.0,\"b\":12356.0}",
                        },
                    },
                ],
            ),
            tool_call_id: None,
        },
        ChatMessage {
            role: "tool",
            content: "{\"result\":17479.0}",
            channel: Some(
                "commentary",
            ),
            recipient: None,
            tool_calls: None,
            tool_call_id: Some(
                "call_0",
            ),
        },
    ],
    tools: [
        Tool {
            type: "function",
            function: Function {
                name: "add",
                description: "adds two numbers",
                parameters: Object {
                    "type": String("object"),
                    "properties": Object {
                        "a": Object {
                            "type": String("number"),
                        },
                        "b": Object {
                            "type": String("number"),
                        },
                    },
                    "required": Array [
                        String("a"),
                        String("b"),
                    ],
                },
            },
        },
    ],
    tool_choice: Some(
        Auto,
    ),
}
ChatCompletionResponse {
    choices: [
        Choice {
            message: ResponseMessage {
                content: "5123\u{202f}+\u{202f}12356\u{202f}=\u{202f}17479.",
                reasoning_content: "We need to respond with the result. The function returned JSON string with result. We should parse it. The result is 17479. So answer: 17479.",
            },
        },
    ],
}

[JouniMi]

I hit the same issue, and from the comment above, I did this (which I have no idea if it's correct or not, I know nothing about llama.cpp internals) which seems to have made it stop crashing at least.

diff --git a/common/chat.cpp b/common/chat.cpp
index 316bd241..ce34484b 100644
--- a/common/chat.cpp
+++ b/common/chat.cpp
@@ -1321,6 +1321,7 @@ static common_chat_params common_chat_params_init_gpt_oss(const common_chat_temp
static void common_chat_parse_gpt_oss(common_chat_msg_parser & builder) {
// TODO @ngxson : this won't work with --special enabled, we should fix that
builder.try_parse_reasoning("<|channel|>analysis<|message|>", "<|start|>assistant<|channel|>final<|message|>");

• builder.add_content(builder.consume_rest());
  if (!builder.syntax().parse_tool_calls) {
  builder.add_content(builder.consume_rest());
  return;
  Then running gpt-oss-120b like this (on revision 1d72c841888b9450916bdd5a9b3274da380f5b36):

CUDA_VISIBLE_DEVICES="0" ./build/bin/llama-server -fa --gpu-layers 99 --threads $(nproc) --threads-batch $(nproc) --batch-size 4096 --ctx-size 131072 --jinja -v --verbose-prompt --model /mnt/nas/models/lmstudio-community/gpt-oss-120b-GGUF/gpt-oss-120b-MXFP4-00001-of-00002.gguf

Seems tool-calling works correctly without crashing then (but without the patch above, it crashes).

Example

I hit the same issue with my tool calling tests, tried this fix which stopped the crashing. Tool-calling seems to be working too, but I am not too confident with the amount of tests done so far.

[Nerexis]

I have created a fix in this fork: https://github.com/Nerexis/llama.cpp/tree/fix/gpt-oss-tool-calls-parser