Compare `ik_llama.cpp`, `vLLM`, `llama.cpp`, and `ktransformers` engines

[ubergarm]

The top runners for hybrid CPU/GPU inferencing of R1 671B currently seem to be:

• ikawrakow/ik_llama.cpp custom llama.cpp fork with many optimizations.
• vLLM - best tensor / data parallel stuff for multi GPU inferencing but very limited CPU / GGUF capabilities
• llama.cpp - good old llama.cpp has some with experimental branches in the works moving slowly along, but refer above to ik_llama.cpp for fastest stuff currently available for some hardware configurations
• ktransformers currently the best option especially combined with AMD Epyc NPS0 but requires at least 16GB VRAM CUDA GPU. Also might be okay for big dual Intel Xeon assuming enough RAM to hold entire model weights twice for data parallel across two sockets, but probably not worth the $$$ afaict so far today

Hey @zts9989 I saw your benchmark over on llama.cpp PR#11397, could you give some more info about your setup? Looks like you have at least one CUDA GPU and probably an AMD processor with NPS0 just guessing? If you like, please compare your speed against ktransformers using this guide.

Also for anyone interested there is a video screen-share of running ktransformers locally by using this guide: https://www.reddit.com/r/LocalLLaMA/comments/1j329e9/ktransformers_troll_rig_r1_671b_udq2_k_xl_on_96gb/

[saood06]

I know you tested ik_llama.cpp in ggml-org/llama.cpp#12088 for one test but in case you didn't know it does offer support for selective offload, flash attention, and even Smart Expert Reduction (tradeoff quality for performance by dynamically reducing expert usage) . I think it is currently far ahead of llama.cpp and it might not be as performant as ktransformers ( haven't seen a direct comparison yet) but is definitely more flexible. It also has it's own custom quant types that are better quality and run time repacking or repacked quants which interleaved rows which can offer substantial performance improvements.

Also on the topic of experts are the numbers you report on the readme for ktransformers with all 8 experts or only 6?

[ubergarm]

@saood06

Oh thanks for your detailed reply on the ik_llama.cpp thread!

it does offer support for selective offload, flash attention, and even Smart Expert Reduction

Sweet! Correct, I ran vanilla llama.cpp commands. Looking forward to giving them a try, FA alone is a win if it allows for both k & v cache quantization. I've used some of the experimental branches on llama.cpp including -ngl 99 -ot=exps, but I never made my own custom MLA quant to try that branch.

I have used --override-kv deepseek2.expert_used_count=int:4 in very early testing, and lower numbers do generate faster (at potential loss in quality). I did notice the ktransformers guys were doing similar things in their published benchmarks running 6. I only run it at the default of 8 experts used now and all my testing/benchmarking is done with default of 8 for llama.cpp and ktransformers.

# ktransformers deepseek v3 config
    num_experts_per_tok (`int`, *optional*, defaults to 8):
        Number of selected experts, None means dense model.

Thanks again for the tips, I'll look into quant packing as well. I might be able to benchmark ktransformers vs llama.cpp@main vs llama.cpp with tensor offload override vs ik_llama.cpp on a 7965WX 24-Cores with 256GB RAM + RTX A6000 48GB VRAM each (same cap=8.6 as 3090). If it looks promising I could try it on my local 9950X 96GB DDR5-6400 and PCIe Gen 5.0 T700 2TB NVMe SSD + 3090TI 24GB VRAM. It gets about 3.5 tok/sec on ktransformers pulling ~5GB/s buffered i/o via mmap() through the page cache lol...

[saood06]

Oh thanks for your detailed reply on the ik_llama.cpp thread!

You’re welcome, I'm glad your testing all of this (and on much nicer hardware than I have access to). I'll reply here for now, but I'd rather continue our conversation in the ik_llama.cpp discussion.

FA alone is a win if it allows for both k & v cache quantization.

Not sure why you care about that, as with MLA the compute buffer actually dominates over the cache size even at 163K context. Also with mla mode 2 there is no V cache needed, ik_llama.cpp also has ways of reducing the compute buffer size, see ikawrakow/ik_llama.cpp#237.

I've used some of the experimental branches on llama.cpp including -ngl 99 -ot=exps, but I never made my own custom MLA quant to try that branch.

I still would recommend making your own quant (and ik_llama.cpp has made it really easy to define a custom quant with this ikawrakow/ik_llama.cpp#244), but there are MLA quants available on huggingface, for example this that work with ik_llama.cpp MLA implementation as well. The benefit of making your own quant is obviously flexibility, being able to use ik_llama exclusive quant types, and using an imatrix (none of the ones I found supporting MLA on huggingface have used an imatrix).

I have used --override-kv deepseek2.expert_used_count=int:4 in very early testing, and lower numbers do generate faster (at potential loss in quality).

I personally don't feel like the tradeoff in quality vs performance is worth it for reducing experts, but ik_llama.cpp's implementation is actually tempting me a bit, read the PR here: ikawrakow/ik_llama.cpp#239

I might be able to benchmark ktransformers vs llama.cpp@main vs llama.cpp with tensor offload override vs ik_llama.cpp on a 7965WX 24-Cores with 256GB RAM + RTX A6000 48GB VRAM each (same cap=8.6 as 3090). If it looks promising I could try it on my local 9950X 96GB DDR5-6400 and PCIe Gen 5.0 T700 2TB NVMe SSD + 3090TI 24GB VRAM. It gets about 3.5 tok/sec on ktransformers pulling ~5GB/s buffered i/o via mmap() through the page cache lol...

Let me know if you do and how things perform. My RPC sync is still a WIP and so I'm CPU only for now as my 3090 is in a different machine than the server with enough RAM for deepseek, but I run deepseek from RAM at about the speed your running it from disk (haven't actually ran it in a while so maybe some optimizations have improved the situation), but my machine is old (but it was cheap).