Need help on downloading model from hugging face and specifying parameters

[gyzerok]

Hello there!

I am now trying to configure Docker Model runner and got couple of questions which I can't figure out from the docs. Hopefully you can lend me a hand here and maybe also improve your documentation.

How can I download specific quant from hugging face? I've tried the following, but it didn't work.

% docker model pull https://huggingface.co/unsloth/Qwen3-32B-GGUF/resolve/main/Qwen3-32B-UD-Q4_K_XL.gguf
Failed to pull model: pulling https://huggingface.co/unsloth/Qwen3-32B-GGUF/resolve/main/Qwen3-32B-UD-Q4_K_XL.gguf failed with status 400 Bad Request: Invalid model reference

% docker model pull hf.co/unsloth/Qwen3-32B-GGUF/resolve/main/Qwen3-32B-UD-Q4_K_XL.gguf
Failed to pull model: pulling hf.co/unsloth/qwen3-32b-gguf/resolve/main/qwen3-32b-ud-q4_k_xl.gguf failed with status 500 Internal Server Error: error while pulling model: reading model from registry: failed to pull model "hf.co/unsloth/qwen3-32b-gguf/resolve/main/qwen3-32b-ud-q4_k_xl.gguf": UNKNOWN - GET https://huggingface.co/v2/unsloth/qwen3-32b-gguf/resolve/main/qwen3-32b-ud-q4_k_xl.gguf/manifests/latest: unexpected status code 404 Not Found: {"error":"Sorry, we can't find the page you are looking for."}

How can I specify additional parameters such as enable_thinking, temperature, top_p and such?

Thank you!

[ilopezluna]

Hello @gyzerok,

You can specify the quantization you're interested in via the tag, for example:
docker model pull hf.co/unsloth/Qwen3-32B-GGUF:IQ1_S

How can I specify additional parameters such as enable_thinking, temperature, top_p and such?

Currently, disabling enable_thinking is not supported, it is always enabled.
Parameters like temperature, top_p, and others can be configured at request time, following the OpenAI API specification:
https://platform.openai.com/docs/api-reference/chat/create

Let me know if you have any other questions!

[gyzerok]

@ilopezluna thank you for the clarifications!

Specifying tags works.

As for parameters - specifying them with the request is only possible when you control the code that is calling the API. In my case, for example I am using Zed, which allows you to configure only temperature for the requests. Since models usually come with recommended parameters it'd be really useful to specify the defaults, I'd like to be aplied if parameter wasn't passed with the request.

[ilopezluna]

Hey @gyzerok I wasn't aware of frameworks or libraries that don't allow defining parameters at request time, thanks for sharing!

We're planning to support configuration at:

• the model artifact level
• startup
• request time

There are ongoing discussions about the implementation details, but we're already working on it, so you can expect an update soon.

[gyzerok]

@ilopezluna it seems like you've released it in 43 version. However I can't get it to work.

Your docs mention --arg=value format which is not how llama-server docs mention it. So either you are giving wrong examples or use your own format. I've tried both with no success. Also when I try too look inside docker model logs I don't see anything related to passing those runtime flags.

The way I know it doesn't work is that Qwen is thinking in my tests.

models:
  qwen3-30b-q8:
    model: hf.co/unsloth/Qwen3-30B-A3B-GGUF:Q8_0
    context_size: 40960
    runtime_flags:
      - '--chat-template-kwargs "{\"enable_thinking\":false}"'
      - '--temperature 0.7'
      - '--top_p 0.8'
      - '--top_k 20'
      - '--min_p 0.01'

What am I missing?

[xenoscopic]

Hey @gyzerok, when the flags are specified as a YAML / JSON array, they're sent directly as argv elements to llama.cpp, so I think you need to break them up a bit - we won't parse the spaces. And since we don't do shell-parsing, the quotation mark escaping would only be necessary in certain scenarios in YAML / JSON. I think you'd want to do something like:

models:
  qwen3-30b-q8:
    model: hf.co/unsloth/Qwen3-30B-A3B-GGUF:Q8_0
    context_size: 40960
    runtime_flags:
      - '--chat-template-kwargs'
      - '{"enable_thinking":false}'
      - '--temperature'
      - '0.7'
      - '--top_p'
      - '0.8'
      - '--top_k'
      - '20'
      - '--min_p'
      - '0.01'

You could also do an inline array (showing here with different quotes / escaping):

models:
  qwen3-30b-q8:
    model: hf.co/unsloth/Qwen3-30B-A3B-GGUF:Q8_0
    context_size: 40960
    runtime_flags: ["--chat-template-kwargs", "{\"enable_thinking\":false}", "--temperature", "0.7", "--top_p", "0.8", "--top_k", "20", "--min_p", "0.01"]

I'm pretty sure you can also do = versions, e.g.

models:
  qwen3-30b-q8:
    model: hf.co/unsloth/Qwen3-30B-A3B-GGUF:Q8_0
    context_size: 40960
    runtime_flags: ["--chat-template-kwargs={\"enable_thinking\":false}", "--temperature=0.7", "--top_p=0.8", "--top_k=20", "--min_p=0.01"]

(Admittedly I haven't had a chance to test any of these manually, but I think they should work).

[xenoscopic]

Also it looks like we don't support --chat-template-kwargs yet - that was just merged upstream in llama.cpp last week, so it hasn't make it through our update pipeline yet.

[ilopezluna]

hi @gyzerok ! In the meanwhile you can use:

--reasoning-budget N                    controls the amount of thinking allowed; currently only one of: -1 for
                                        unrestricted thinking budget, or 0 to disable thinking (default: -1)
                                        (env: LLAMA_ARG_THINK_BUDGET)

BTW --temperature is not a valid argument, use --temp N instead, you can see here the available arguments.
I also recommend you to use --verbose to validate you are using the correct arguments.
Let us know how it goes 🙏

[kiview]

Gonna close this, since I assume the last comments resolved the issues.
Please let us know if issues persist and we can reopen @gyzerok.