Eval bug: output from gpt-oss-20b-F16 model has no comparison the output of the same prompt on gpt-oss.com

[jgforbes]

Name and Version

$ ./llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
version: 6121 (e54d41b)
built with cc (GCC) 14.3.1 20250523 (Red Hat 14.3.1-1) for x86_64-redhat-linux

Sending the simple prompt "What is bugonia?" to the 20b model on gpt-oss.com gives a perfect response.
With llama-cli it tries to reason an answer but never comes close the the correct answer from gpt-oss.com.

Neither of these invocations give an acceptable anser:
$ ./llama.cpp/llama-cli -hf unsloth/gpt-oss-20b-GGUF:F16 --jinja -ngl 99 --threads -1 --ctx-size 16384 --temp 1.0 --top-p 1.0 --top-k 0

$ ./llama.cpp/llama-cli -hf ggml-org/gpt-oss-20b-GGUF -c 0 -fa --jinja --reasoning-format none

Operating systems

Linux

GGML backends

CUDA

Hardware

Intel(R) Core(TM) i7-5820K CPU
RTX 3090

Models

No response

Problem description & steps to reproduce

run command above with given prompt

First Bad Commit

New with this model

Relevant log output

NA

[victorb]

Sending the simple prompt "What is bugonia?" to the 20b model on gpt-oss.com gives a perfect response.

For reference, what is the "perfect response"? On gpt-oss.com, I get "“Bugonia” isn’t a real place on earth – it’s almost always a fictitious or brand‑specific name." + a bunch of guesses what it could be + asking me to give additional context. Is that the correct one?

[steampunque]

"Bugonia" is the current state of gpt-oss-20b functionality on Llama.cpp. See #15155.

BEAMS=2 BEAM_TOKENS=256 lm Give a definition of the term "bugonia"
<|channel|>analysis<|message|>The user asks: "Give a definition of the term 'bugonia'". We need to provide a definition. But "bugonia" is not a standard English word. It might be a made-up term. Could be a portmanteau of "bug" and "agonia" meaning suffering. Could be a fictional concept. The user likely wants a definition. We can provide a creative definition: "Bugonia: a state of extreme frustration caused by persistent software bugs." Or "Bugonia: a hypothetical planet inhabited by bugs." Or "Bugonia: a term used in software development to describe the phenomenon where a bug causes cascading failures." Or "Bugonia: a comedic term for the feeling of being overwhelmed by bugs in a codebase." The user didn't specify context. So we can give a general definition: "Bugonia is a term used to describe a situation or environment where software bugs are rampant and cause significant problems." Or "Bugonia: a fictional land of bugs." We can also mention that it's not a standard dictionary word. Provide a definition and possible usage. Let's do that.<|start|>assistant<|channel|>final<|message|>Bugonia (noun)
An informal, often humorous, term used to describe a situation, environment, or state of affairs in which software bugs are so numerous, persistent, or disruptive that they dominate the experience or operation of a system. In other words, it’s the “bug‑infested” version of a normal working state.

Typical contexts

Context	Example usage
Software development	“After the last release, the codebase fell into bugonia; every feature was breaking.”
Project management	“We’re in bugonia right now—every sprint is spent firefighting rather than delivering new functionality.”
Tech humor	“If you’ve ever stared at a stack trace for hours, you’ve lived in bugonia.”
Fictional world‑building	“In the novel, Bugonia is a sprawling metropolis of sentient insects that thrive on human error.”

Etymology
A blend of bug (the common term for a software defect) and the suffix ‑onia (as in mania, phobia, agonia), suggesting a state or condition. The word is not found in standard dictionaries; it is a neologism that has circulated in tech‑centric online communities and informal writing.

Synonyms / related terms

• Bug‑storm
• Glitch‑land
• Error‑zone
• Crash‑city

Note
Because “bugonia” is not a formal lexical entry, its meaning can vary with context. In most cases it carries a mildly negative, sometimes humorous connotation, implying that bugs have taken over or are overwhelming.

[jgforbes]

https://en.wikipedia.org/wiki/Bugonia#

In the ancient Mediterranean region, bugonia or bougonia was a ritual based on the belief that bees were spontaneously (equivocally) generated from a cow's carcass, although it is possible that the ritual had more currency as a poetic and learned trope than as an actual practice.

And it goes on and on about this ancient belief.

I ran the same prompt multiple times on gpt-oss.com and never got a response like @victorb. I also ran it on a deployment using vLLM and it returned similar content about spontaneous generation of bees from an animal carcass and never the random guesses.

I use this simple prompt to test LLMs as it is esoteric information, but has been on wikipedia for a long time. Many LLMs are unable to provide information on this term.......

[jgforbes]

I tried the prompt on gpt-oss.com again and I got an answer similar to @victorb a couple of times.

[abc-nix]

Are you sure you selected the GPT-OSS 20B model on the gpt-oss.com website?

I get similar (wrong) results for both the online and local model with medium reasoning effort.

If I go to the 120B model, I get the correct answer in both, with similar wording in the online and local execution (everything with llama.cpp with the ggml model, but the unsloth jinja, and the #15181 harmony parsin patch, and same parameters).

gpt-oss 20B medium reasoning comparison

gpt-oss 120B medium reasoning comparison

[jgforbes]

@abc-nix Good question. When you change models on gpt-oss.com, you also have to clear the chat to get a fresh start. If you ask the question on the 120b model it gives a long answer with the expected content. If you switch to the 20b model without clearing the chat it gives a briefer answer related to the answer provided by the 120b model. Clearing the chat and the 20b model gives the nonsense reasoning answers as shown above.

I was able to get the 120b gpt-oss model to work on a RTX 3090 with llama.cpp and -ngl 11. The speed is acceptable and it gives answers similar to the 120b model on gpt-oss.com.

[abc-nix]

@jgforbes No problem. I also saw that the site keeps the older context, and that is why it correctly labels the conversation based on the previous conversion.

For GPT-OSS 120B on a single 3090, try using -fa -ngl 99 -ub 1024 --n-cpu-moe 25 instead for faster inference and Prompt Processing. It will be much better than limiting full layer offload only. You may need to increase the --n-cpu-moe number if you require a larger context, but it will still run faster than limiting the -ngl number.