Feature Request: support intel amx for further accelerate

[zhaoyukoon]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I learned from ktransformers-Intel-AMX that amx instruction can further improve inference speed for MoE models.

Is there any plan to support amx in ik_llama? Thanks!

Motivation

Ktransformer kernel can achieve 21 TFLOPS of BF16 throughput and 35 TOPS of Int8 throughput on Xeon4 CPUs — about 4× faster than PyTorch’s general AMX kernel. For DeepSeek-V3, pairing a Xeon4 CPU with a single RTX 4090 GPU achieves 418 tokens/s end-to-end throughput, close to the performance of multi-machine, multi-GPU setups. KTransformers’ AMX kernel is the first AMX kernel specifically designed for MoE inference scenarios, significantly lowering the hardware barrier for large model deployment and enabling more developers to enjoy GPU cluster level inference experiences at lower cost.

Possible Implementation

No response

[ikawrakow]

If someone gives me access to a system with AMX support, then sure, I would work on that.

But out of curiosity, do you have a performance comparison between ik_llama.cpp and KTransformers on the same system?

[zhaoyukoon]

If someone gives me access to a system with AMX support, then sure, I would work on that.

But out of curiosity, do you have a performance comparison between ik_llama.cpp and KTransformers on the same system?

I can access a server equipped with AMX Intel CPUs, however I have no permission to add other uses. I can help to run test on this server.

I tested ktransformers on another AMD server with 24GB 4090D, which can get 15+ tokens/s decoding speed. I have not tested ik_llama yet, I learned that llama.cpp can get 7 tokens/s on pure CPU.

https://github.com/XuanwuLab/llama.cpp_deepseek/blob/main/llama-mmap.cpp

https://mp.weixin.qq.com/s/vIrvbVJ6Nv00Ehre1zZwMw [In Chinese]

[ikawrakow]

I cannot say that I'm particularly impressed with the performance reported in ktransformers-Intel-AMX. For convenience here is what they report:

My system is Ryzen-7950X CPU + 4080 GPU. Based on benchmarks from here and here, my CPU is only marginally faster than their "consumer" level system with Intel-14900KF + 4090 GPU. I don't have enough RAM to run Qwen3-235-A22B, but here is what I get for Qwen3-30B-A3B quantized with IQ4_XS (so corresponds to their 4-bit result) with ik_llama.cpp:

CPU only

model	size	backend	test	t/s
qwen3moe 30B IQ4_XS	15.24 GiB	CPU	pp512	480.78 ± 2.11
qwen3moe 30B IQ4_XS	15.24 GiB	CPU	tg128	29.17 ± 0.08

Here pp512 corresponds to what they call "prefill" and tg128 is what they call "decode". So, even without a GPU ik_llama.cpp beets their prefill performance by 2X, and is faster than their "4-way decode" performance on the "consumer" level system that has roughly the same speed as mine.

CPU+GPU

Here speed depends on how many layers I offload to the GPU. But let's keep 18 layers on the CPU so I have enough VRAM for the maximum context of 41,000 tokens on my paltry 16 GB GPU. Here is what I get with that:

model	size	backend	test	t/s
qwen3moe 30B IQ4_XS	15.24 GiB	CUDA	pp2048	3039.84 ± 24.96
qwen3moe 30B IQ4_XS	15.24 GiB	CUDA	tg128	77.44 ± 0.39

So, 15X their prefill performance and 3X their "4-way decode" performance ("consumer level" system), and 8.7X prefill, 1.5X "4-way decode" (Xeon 4 workstation).

[ikawrakow]

I can access a server equipped with AMX Intel CPUs, however I have no permission to add other uses. I can help to run test on this server.

This will be way too tedious. I have to build with AMX instructions enabled, then you test and find gibberish, then I second-guess where is the bug, change something, you test and find gibberish, rinse and repeat. I have to have access to the AMX-enabled system while writing the code.

[zhaoyukoon]

I can access a server equipped with AMX Intel CPUs, however I have no permission to add other uses. I can help to run test on this server.

This will be way too tedious. I have to build with AMX instructions enabled, then you test and find gibberish, then I second-guess where is the bug, change something, you test and find gibberish, rinse and repeat. I have to have access to the AMX-enabled system while writing the code.

Do you have any requirements on CPU and memory for development? Is server with 16 vCPU AMX and 32GB enough?

[ikawrakow]

Do you have any requirements on CPU and memory for development? Is server with 16 vCPU AMX and 32GB enough?

Yes, that's should be enough for development.

But before you go and rent a cloud instance, let's start by you fist testing ik_llama.cpp on your system and comparing performance to KTransformers.

Let's also make sure that the expectations are aligned:

• It is extremely unlikely AMX will improve token generation (TG) speed
• It is very unlikely AMX will improve prefill speed for hybrid CPU/GPU inference for most models. Only the LLaMA-4 models may get faster
• AMX will improve prefill performance for CPU-only inference compared to vanilla AVX2 implementations such as what you have in llama.cpp or KTransformers. If it will improve performance compared to the existing ik_llama.cpp implementation remains to be seen.

[kirnat]

While I’d be excited to see AMX support, I can’t say the kTransformers Qwen3 benchmark proves its usefulness. I can’t verify the pp/tg window sizes or the exact model they used, but as an inexact comparison, I got the below results in ik_llama for Qwen3 235B with Xeon 8480 (ES), 8-channel 4800MT DDR5 and a blackwell GPU.

Model used:
unsloth/Qwen3-235B-A22B-GGUF/UD-Q4_K_XL/Qwen3-235B-A22B-UD-Q4_K_XL

size	params	backend	ngl	threads	n_batch	n_ubatch	fa	rtr	fmoe	test	t/s
124.91 GiB	235.09 B	CUDA	93	52	8192	8192	1	1	1	pp2048	192.02 ± 0.06
124.91 GiB	235.09 B	CUDA	93	52	8192	8192	1	1	1	pp16384	185.33 ± 0.34
124.91 GiB	235.09 B	CUDA	93	52	8192	8192	1	1	1	tg512	18.74 ± 0.02
124.91 GiB	235.09 B	CUDA	93	52	8192	8192	1	1	1	tg2048	18.58 ± 0.03

The 30B model performs really well on CPU only, below is with GPU hidden.

Model used:
unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-UD-Q4_K_XL

size	params	backend	ngl	threads	fa	fmoe	test	t/s
16.49 GiB	30.53 B	CUDA	0	32	1	1	pp512	510.65 ± 2.49
16.49 GiB	30.53 B	CUDA	0	32	1	1	pp2048	454.62 ± 0.18
16.49 GiB	30.53 B	CUDA	0	32	1	1	tg128	69.77 ± 0.02
16.49 GiB	30.53 B	CUDA	0	32	1	1	tg512	69.15 ± 0.01

Thanks a lot for the impressive work ikawrakow!

[ikawrakow]

Has anyone tried mainline llama.cpp AMX implementation?

[zhaoyukoon]

Has anyone tried mainline llama.cpp AMX implementation?

ggml-org/llama.cpp#12003

It seems that llama.cpp supports amx

[ikawrakow]

It seems that llama.cpp supports amx.

That's why I asked if somebody has tried. It would be even more interesting if someone has compared llama.cpp performance to ik_llama.cpp on an AMX CPU.

[kirnat]

Confirming AMX buffer

llama.cpp/build/bin/llama-cli -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: CPU_Mapped model buffer size = 4685.30 MiB
load_tensors: AMX model buffer size = 4491.48 MiB
........................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: freq_base = 500000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context: CPU output buffer size = 0.49 MiB
llama_kv_cache_unified: kv_size = 4096, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1, padding = 32
llama_kv_cache_unified: CPU KV buffer size = 512.00 MiB
llama_kv_cache_unified: KV self size = 512.00 MiB, K (f16): 256.00 MiB, V (f16): 256.00 MiB
llama_context: CPU compute buffer size = 296.01 MiB

llama.cpp bench

llama.cpp/build/bin/llama-bench -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -t 52 -fa 1

model	size	params	backend	threads	fa	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CPU	52	1	pp512	228.18 ± 0.03
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CPU	52	1	tg128	37.28 ± 0.01

build: e3a9421b (5389)

ik_llama bench

ik_llama.cpp/build/bin/llama-bench -ngl 0 -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -t 52 -fa 1
ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected

model	size	params	backend	ngl	threads	fa	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CUDA	0	52	1	pp512	348.00 ± 0.43
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CUDA	0	52	1	tg128	42.48 ± 0.03

build: 2ec2229 (3702)

---

Let me know if you want me to test with another model specific settings. I used high thread count since it helps prompt processing while penalizes token generation slightly, but not too much in this case.

[ikawrakow]

Thanks!

You could try adding -rtr 1 to the ik_llama.cpp benchmark run. This normally gives a significant boost in PP performance.

[kirnat]

I hadn't even considered it for CPU only inference. I have used it alot day to day for hybrid inference with great results.

Same settings as above and GPU hidden but with rtr enabled.

model	size	params	backend	ngl	threads	fa	rtr	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CUDA	0	52	1	1	pp512	444.89 ± 0.96
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CUDA	0	52	1	1	pp16384	267.07 ± 3.60
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CUDA	0	52	1	1	tg128	43.21 ± 0.03
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	CUDA	0	52	1	1	tg2048	41.39 ± 0.01

Still amazed how relatively low the slow down is in ik at larger context sizes. This tranlates to Qwen3, DeepSeek v3 and Llama 4 Maverick as well.

[ikawrakow]

So, ik_llama.cpp without AMX is nearly two times faster than llama.cpp with AMX.