Model runner no longer outputs reasoning tokens?

[f2bo]

I notice that model runner no longer outputs the reasoning content (i.e. the <think>...</think> block) for models such as qwen3 or deepseek-r1-distill-llama. I started noticing this following one of the more recent updates, since previously I did see the models output this type of content.

For example, below is the output from Qwen3 and DeepSeek from Model Runner.

Qwen3

docker model run ai/qwen3
Interactive chat mode started. Type '/bye' to exit.
> How many Rs in strawberry?
The word **"strawberry"** contains **three** instances of the letter **R**.

Here's the breakdown:
**S-T-R-A-W-B-E-R-R-Y**
- **R** at position 3
- **R** at position 8
- **R** at position 9

**Answer:** 3 R's.

DeepSeek-R1-Distill-Llama

docker model run ai/deepseek-r1-distill-llama
Interactive chat mode started. Type '/bye' to exit.
> How many Rs in strawberry?
The word "strawberry" contains three Rs.

**Step-by-Step Explanation:**

1. **Write Out the Word:** S-T-R-A-W-B-E-R-R-Y.
2. **Identify Each Letter:** S, T, R, A, W, B, E, R, R, Y.
3. **Count the Rs:**
   - The third letter is R (1).
   - The eighth letter is R (2).
   - The ninth letter is R (3).
4. **Total Count:** 3 Rs.

**Answer:** There are three Rs in "strawberry."

For contrast, this is the output from Qwen3 running in Ollama.

Qwen3 (in Ollama)

ollama run qwen3
>>> How many Rs in strawberry?
Thinking...
Okay, the user is asking how many times the letter 'R' appears in the word "strawberry." Let me break this down
step by step.

First, I should write out the word clearly: S-T-R-A-W-B-E-R-R-Y. Wait, let me make sure I have the spelling right.
Strawberry is spelled S-T-R-A-W-B-E-R-R-Y. Let me count each letter one by one.

Starting with the first letter: S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). Wait,
that's 10 letters. Let me check again. S-T-R-A-W-B-E-R-R-Y. Yes, that's correct. So the letters are S, T, R, A, W,
B, E, R, R, Y.

Now, I need to count how many 'R's are there. Let's go through each letter:

1. S – not R
2. T – not R
3. R – that's one
4. A – not R
5. W – not R
6. B – not R
7. E – not R
8. R – that's two
9. R – that's three
10. Y – not R

So there are three 'R's in "strawberry." Wait, but let me double-check. Sometimes people might miscount. Let me
write them out again:

Letters in strawberry: S, T, R, A, W, B, E, R, R, Y. So positions 3, 8, and 9 are R. That's three Rs.

Hmm, but I've heard before that sometimes people might miss a letter. Let me confirm once more. The word is
spelled S-T-R-A-W-B-E-R-R-Y. Yes, the third letter is R, then after the E, there are two more Rs. So three Rs in
total.

I think that's correct. So the answer should be 3.
...done thinking.

The word **strawberry** contains **3** instances of the letter **R**.

**Breakdown:**
S-T-R-A-W-B-E-R-R-Y
Positions: 3rd, 8th, and 9th letters are **R**.

**Answer:** 3 Rs.

Also, inferencing seems to be significantly slower after the latest updates, but this could be subjective since I didn't pay attention to this aspect previously. Qwen3 is certainly a lot slower (by more than 100%) than this model running in Ollama but again, I'm not certain if both models are exactly the same.

https://hub.docker.com/r/ai/qwen3

docker model inspect ai/qwen3
{
    "id": "sha256:79fa56c07429f64f41950fe2f524937cf3ae9ea9bd3d7ada72170b036ea3cc85",
    "tags": [
        "ai/qwen3"
    ],
    "created": 1746022478,
    "config": {
        "format": "gguf",
        "quantization": "IQ2_XXS/Q4_K_M",
        "parameters": "8.19 B",
        "architecture": "qwen3",
        "size": "4.68 GiB"
    }
}

https://ollama.com/library/qwen3:8b

ollama show qwen3
  Model
    architecture        qwen3
    parameters          8.2B
    context length      40960
    embedding length    4096
    quantization        Q4_K_M

  Capabilities
    completion
    tools
    thinking

  Parameters
    top_k             20
    top_p             0.95
    repeat_penalty    1
    stop              "<|im_start|>"
    stop              "<|im_end|>"
    temperature       0.6

  License
    Apache License
    Version 2.0, January 2004

[p1-0tr]

Hi, thanks for bringing this to our attention, and sorry for the delay in replying. We should have a fix for printing reasoning tokens on the CLI released soon.

As to the performance issues you noticed - is the problem reproducible? What hardware are you using? I've been able to repro similar behaviour, but it boiled down to the model doing a lot more reasoning from time to time (this took around 4x the "normal" time in my case). This would happen with both DMR and ollama.

[f2bo]

Hi @p1-0tr. Thanks for your response.

We should have a fix for printing reasoning tokens on the CLI released soon.

That's good to hear. Just in case, I should mention that they are not only missing from the CLI, but also when accessing a model through the OpenAI endpoint.

As to the performance issues you noticed - is the problem reproducible? What hardware are you using? I've been able to repro similar behaviour, but it boiled down to the model doing a lot more reasoning from time to time (this took around 4x the "normal" time in my case). This would happen with both DMR and ollama.

Yes. It is reproducible.

I'm running on CPU on a Windows 11 Hyper-V virtual machine that is currently assigned 48GB of memory. Docker is running in this VM and Ollama is also running as a Docker container in this same VM.

Since I don't get to see the reasoning output, I'm currently unable to tell how much reasoning is done with the models I mentioned, so instead I pulled llama3.2 (3B) and tested both in DMR and Ollama. The difference in speed is already evident using the CLI. With this model, Ollama produces output at approximately my reading speed, whereas I can read faster than DMR outputs its tokens.

Running ollama run <model> --verbose parameter will output timings for a response, but I couldn't find an equivalent parameter for model run <model>. I see that there's a --debug argument but it doesn't seem to make any visible difference. I thought perhaps that it might write additional information to the log, but now I notice that the last entry in my log is from a month ago and it doesn't seem to be updating any more, so that's another problem. Perhaps you can suggest how I can restore the logging.

Instead, I ran a few tests with my own application that I use as a test harness and that accesses Ollama and DMR through their OpenAI endpoints. I can't guarantee that the measurement of tokens/sec is done correctly since I only use it as a rough indicator for my own comparisons, but it does reflect the relative speed difference between both models.

Here's the output for both tests.

Docker Model Runner

Ollama

I haven't used Model Runner extensively, but as I mentioned, I don't seem to recall such a difference in speed in the initial releases. I seem to have several unused docker/model-runner images in Docker which I assume are from earlier releases and that I haven't deleted yet. Is there a way to temporarily run the CLI with one of these images so that I can check?

[p1-0tr]

Thanks :)

Just in case, I should mention that they are not only missing from the CLI, but also when accessing a model through the OpenAI endpoint.

Hmm, that is concerning. A few questions to help me figure out what's going on. Do you mean that the completion responses are missing the reasoning_content field?

In my case it is there:

curl http://localhost:12434/engines/v1/chat/completions -d '{
  "model": "ai/qwen3",
  "messages": [
    {"role": "user", "content": "What is 1+1?"}
  ]}' | jq .
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": "Okay, the user is asking \"What is 1+1?\" Let me start by recalling basic arithmetic. In mathematics, addition is a fundamental operation. When you add 1 and 1 together, the result should be 2. But maybe I should check if there's any alternative interpretation or context where this might not hold.\n\nWait, could there be a scenario where 1+1 isn't 2? For example, in some non-standard number systems or under different mathematical structures? Let me think. In standard arithmetic, 1+1 is definitely 2. However, in certain contexts like modular arithmetic, if we're working modulo 2, then 1 + 1 equals 0. But the question doesn't specify any such context. So unless stated otherwise, the default is standard arithmetic.\n\nAnother angle: maybe the user is testing if I know about binary numbers? In binary, 1 + 1 is 10, which is 2 in decimal. But again, the question is straightforward, so it's more likely they want the simple answer.\n\nCould there be any trick or play on words here? For instance, in some languages, \"1+1\" might have a different meaning, but in English, it's a direct arithmetic question. The user might also be checking if I understand the basics, which is important for more complex topics.\n\nI should also consider if the user is a child learning math, in which case the answer is simply 2. Or maybe they're a student who needs confirmation. Either way, the answer remains the same. It's essential to state the answer clearly and confirm that it's correct in the standard context. If there's any ambiguity, I should mention it but emphasize the standard result.",
        "content": "The result of $1 + 1$ in standard arithmetic is **2**. \n\nIn mathematics, addition of the numbers 1 and 1 follows the basic rules of arithmetic, yielding the sum 2. If there is a specific context (e.g., modular arithmetic, binary operations, or abstract algebra), the answer might differ, but without such context, the standard answer is **2**. \n\n**Answer:** 2."
      }
    }
  ],
  "created": 1755072950,
  "model": "ai/qwen3",
  "system_fingerprint": "b1-82bf586",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 448,
    "prompt_tokens": 15,
    "total_tokens": 463
  },
  "id": "chatcmpl-ItO8Vq97d9AXjAJbcADsUn4vxsncHKYM",
  "timings": {
    "prompt_n": 1,
    "prompt_ms": 54.984,
    "prompt_per_token_ms": 54.984,
    "prompt_per_second": 18.187108977156992,
    "predicted_n": 448,
    "predicted_ms": 6610.582,
    "predicted_per_token_ms": 14.755763392857144,
    "predicted_per_second": 67.77012977072215
  }
}

Or do you expect the reasoning to be included in content? In that case that currently could be done via Compose (https://docs.docker.com/ai/compose/models-and-compose/) by adding --reasoning-format=none in runtime_flags:. We are working on improving available runner configuration, so that will become easier to do in upcoming releases.

Yes. It is reproducible.

Ok, that's definitely different to the variability I saw.

I'm running on CPU on a Windows 11 Hyper-V virtual machine that is currently assigned 48GB of memory. Docker is running in this VM and Ollama is also running as a Docker container in this same VM.
I seem to have several unused docker/model-runner images in Docker which I assume are from earlier releases and that I haven't deleted yet. Is there a way to temporarily run the CLI with one of these images so that I can check?

One more question - are you using Docker Desktop or Docker CE installed inside WSL? In the latter case you should be able to use an environment variable - MODEL_RUNNER_CONTROLLER_VERSION - to select an older version (e.g. v0.0.8 is the previous version). This will not work with Docker Desktop, though. Also if you are using Docker Desktop you should not need the docker/model-runner images (so them being there could indicate a bug in the model CLI).

[f2bo]

Do you mean that the completion responses are missing the reasoning_content field?

No. You're right. The responses do include the reasoning_content field.

Or do you expect the reasoning to be included in content?

Yes, or rather, it's the behavior I saw when I first started testing Model Runner. It's also how Ollama crafts its responses where the reasoning text is included within the content delimited by <think>...</think> tags.

I'm currently accessing the model using the Microsoft.Extensions.AI library and it does not (yet?) publicly expose a mechanism for retrieving the "reasoning_content" separately so all I saw was the "regular" content. Since the CLI was also missing it, I incorrectly assumed it was also missing from the API response. I did find some indication that it should be addressed at some point (dotnet/extensions#6208 and openai/openai-dotnet#259). In any case, as long as the reasoning content is there, it should be solvable. I will look into the --reasoning-format Compose workaround you mention.

are you using Docker Desktop or Docker CE installed inside WSL?

I'm using Docker Desktop.

Also if you are using Docker Desktop you should not need the docker/model-runner images (so them being there could indicate a bug in the model CLI).

It could be related to a problem I had with Model Runner some time ago following an update where I started getting the following error whenever I used the CLI. For example, this is the output I recorded at the time:

$ docker model version
Docker Model Runner version v0.1.23
Docker Engine Kind: Moby

$ docker model run ai/qwen3
7ae945c85670: Pull complete Digest: sha256:2b06fa65d2c72af24838d07ebaa93e248ed9d1cadbabb68f0a784d038841fd9f
Status: Downloaded newer image for docker/model-runner:latest

Successfully pulled docker/model-runner:latest
unable to initialize standalone model runner: unable to initialize standalone model runner container: failed to create container docker-model-runner: Error response from daemon: Conflict. The container name "/docker-model-runner" is already in use by container "7605a042914581f069ced6c6083cb9cdbe57e961f69a4fe4c8c3b28698e252c4". You have to remove (or rename) that container to be able to reuse that name.

I was able to temporarily fix the problem by removing the container, but then the problem would reappear whenever I used the CLI. Strangely, I could use the models without issue from the Docker Desktop UI. I finally removed all the models that I had pulled, ran uninstall-runner followed by install-runner and pulled the models again, and I haven't had this problem since. However, I did find that I still had some model files stored in WSL under the Docker volumes folder that apparently weren't being used and that weren't removed when I previously rm'd the models. Since Model Runner now seems to store models in the user's profile folder at .docker\models, I eventually removed them after determining that they weren't being used. Was there a change in the model storage location from earlier versions?

In any case, my remaining problem now is the speed issue, which is not insignificant. Do you think this is something that can be resolved?

[f2bo]

I just noticed from the information I pasted above that the output from docker model version at that time was showing Moby as the engine kind:

$ docker model version
Docker Model Runner version v0.1.23
Docker Engine Kind: Moby

Whereas, now it is Docker Desktop:

$ docker model version
Docker Model Runner version v0.1.36
Docker Engine Kind: Docker Desktop

Not sure why this happened, but I suspect it could be relevant.

[p1-0tr]

I eventually removed them after determining that they weren't being used. Was there a change in the model storage location from earlier versions?

No, the Docker Desktop variant of model runner always used .docker\models to store models. The volumes must have been created because of the same bug (the model CLI misidentifying DD as CE) that caused the docker/model-runner images to be downloaded.

In any case, my remaining problem now is the speed issue, which is not insignificant. Do you think this is something that can be resolved?

Yes, I have a fairly strong suspicion as to what the root cause may be. I'll let you know once the potential fix becomes available.

[f2bo]

Yes, I have a fairly strong suspicion as to what the root cause may be. I'll let you know once the potential fix becomes available.

Great. Looking forward to your findings. Thanks!

[p1-0tr]

I've published a new build of the llama.cpp server we use for model runner. We were still shipping a relatively "safe" (in terms of CPU compatibility) build on x86 Windows. Now it should be aligned to what we do on x86 Linux, and leverage llama.cpps ability to pick the CPU backend with optimisations best suited to the CPU that one is using.

If you quit and re-open Docker Desktop that will trigger model runner to look for new versions of the llama-server image, and update if necessary. Once the update is complete ~\.docker\inference\bin should contain:

com.docker.llama-server.exe
com.docker.nv-gpu-info.exe
ggml-base.dll
ggml-cpu-alderlake.dll
ggml-cpu-haswell.dll
ggml-cpu-icelake.dll
ggml-cpu-sandybridge.dll
ggml-cpu-sapphirerapids.dll
ggml-cpu-skylakex.dll
ggml-cpu-sse42.dll
ggml-cpu-x64.dll
ggml.dll
llama-mtmd-cli.exe
llama.dll
mtmd.dll

instead of:

com.docker.llama-server.exe
com.docker.nv-gpu-info.exe
ggml-base.dll
ggml-cpu.dll
ggml.dll
llama-mtmd-cli.exe
llama.dll
mtmd.dll

[f2bo]

Yes! Now the performance is comparable to Ollama's.

Thank you again for your help.

[p1-0tr]

Thanks for bringing the issues to our attention :)

[kiview]

Issue mentioned in the opening post fixed by docker/model-cli#137.