DeepSeek-R1-Distill-Qwen-32B-GGUF needs the deepseek-r1-qwen tokenizer

[Kenshiro-28]

Current Behavior

Can't run DeepSeek-R1-Distill-Qwen-32B-GGUF, I get a tokenizer error:

"llama_model_load: error loading model: error loading model vocabulary: unknown pre-tokenizer type: 'deepseek-r1-qwen'"

Model url: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF

Environment and Context

Hardware: VPS with 32 GB RAM
OS: Debian 12

Using the current version of llama-cpp-python

Suggestion

Upgrade to the required version of llama.cpp:

https://github.com/ggerganov/llama.cpp/releases/tag/b4514

[Mrw33554432]

I guess it can be solved by simply change a commit id? Check c9dfad4.

[Mrw33554432]

I guess it can be solved by simply change a commit id? Check c9dfad4.

I change it and when i run deepseek R1 model it show: AttributeError: /root/llama-cpp-python/llama_cpp/lib/libllama.so: undefined symbol: llama_rope_type It seems that the the llama.cpp API version not match the version for the version of llama-cpp-python.

Then we will need to wait for updates.

I guess it won't be too hard to match up api, perhaps we can just pass the code to GPT and expect it to solve the issue.

Would make a try if I get some time (busy week).

[Kenshiro-28]

Thank you!

[JamePeng]

It is not just a commit synchronization issue. There are many LLAMA_API parts in llama_cpp.py that need to be updated and synchronized to the new version refactored in llama.h of llama.cpp master.

[rzafiamy]

I am eager to receive the update as well. The Deepseek R1 model is truly remarkable. Hope this adaptation of API could be more automatized through AI coder in future

[lexasub]

does llama.cpp have such problem?

[Kenshiro-28]

I guess using llama.cpp directly works fine as bartowsky did the quant with it, check the model url.

[Alchete]

"Deepseek R1 is one of the most amazing and impressive breakthroughs I’ve ever seen — and as open source, a profound gift to the world." - Marc Andreessen

Is it a big ask to add support for this model?

[JamePeng]

@Kenshiro-28 You can try my repaired version. #1901

[davidmroth]

I guess using llama.cpp directly works fine as bartowsky did the quant with it, check the model url.

Yes, using llama.cpp directly works fine. Tested and verified.

[davidmroth]

@Kenshiro-28 You can try my repaired version. #1901

Confirmed working! Great job @JamePeng!!

[Kenshiro-28]

@Kenshiro-28 You can try my repaired version. #1901

@JamePeng Thank you! I will wait for the official release :)

[Kenshiro-28]

Working with version 0.3.7, you can install it with:

pip install llama-cpp-python