CPU beating GPU in token generation speed

[ikawrakow]

The TriLM ternary models are available in various sizes, so I was curious to look into prompt processing (PP) and token generation (TG) speed when the model is small enough to fit in the CPU cache. I have a Ryzen-7950X CPU with 64 MiB of L3 cache, and the 99M parameter TriLM model is 46 MiB when quantized with IQ2_TN. So, without further ado, lets look at a comparison between the Ryzen-7950X and an RTX-4080 in this case:

backend	threads	test	t/s
Ryzen-7950X	16	pp1500	8268.11 ± 48.34
Ryzen-7950X	4	tg500	1016.65 ± 22.17
Ryzen-7950X	8	tg500	1224.83 ± 32.28
Ryzen-7950X	16	tg500	1240.54 ± 25.74
RTX-4080	-	pp1500	110388 ± 250
RTX-4080	-	tg500	1136.64 ± 4.99

The GPU is still much faster than the CPU for prompt processing (although the difference, which is typically a factor of ~30 between this specific GPU and CPU, has shrunk to just a factor of 13), but now the CPU beats the GPU in TG speed!

I also have an M2-Max laptop (the version with a 30-core GPU). Here is what we get:

backend	threads	test	t/s
M2-Max CPU	8	pp1500	5209.27 ± 21.48
M2-Max CPU	2	tg500	692.87 ± 1.74
M2-Max CPU	4	tg500	841.48 ± 5.96
M2-Max CPU	8	tg500	894.73 ± 10.03
M2-Max GPU	4	pp1500	25824 ± 562
M2-Max GPU	4	tg500	464.86 ± 3.85

Also here the GPU is faster for PP (but just 5X faster), but the CPU wipes the floor with the GPU for TG, beating it close to 2X using all 8 threads, and 1.5X with just 2 threads!

[ikawrakow]

Now that we have efficient Flash Attention (FA) implementation on the CPU via PR #32, we can compare again performance between the CPU and GPU for this tiny 99M parameter model. We get

model	size	params	backend	ngl	threads	fa	test	t/s
IQ2_BN - 2.06 bpw TriLM	45.89 MiB	99.76 M	CUDA	100	1	1	pp1500	156827.38 ± 727
IQ2_BN - 2.06 bpw TriLM	45.89 MiB	99.76 M	CUDA	100	1	1	tg500	1496.37 ± 36.79
IQ2_BN - 2.06 bpw TriLM	45.89 MiB	99.76 M	CPU	0	16	1	pp1500	12133.80 ± 51.45
IQ2_BN - 2.06 bpw TriLM	45.89 MiB	99.76 M	CPU	0	16	1	tg500	1509.52 ± 9.65

TG speed is now about the same, which is still quite remarkable.

FA has improved CPU prompt processing speed by almost 50%, TG by 22%.

[saood06]

Is there a chance SpargeAttn could be implemented here. Code here, Paper here.

If it could would it benefit speed on CPU?

[ikawrakow]

Other than the paper, is there any evidence that this works as advertised? If I did nothing else but implementing breakthroughs announced on arXiv, the day still wouldn't have enough hours.

[saood06]

Other than the paper, is there any evidence that this works as advertised?

Not really (there are multiple ComfyUI custom nodes that port support but not much on people using it), the paper looked interesting to me and the idea makes sense to me, but the implementation they have looks premature. The same group put out SageAttention/SageAttention2 which has been widely adopted (mostly for image/video models) and the performance matched the paper but SpargeAttn has gotten interest but not much adoption because of the state of the implmentation.

If I did nothing else but implementing breakthroughs announced on arXiv, the day still wouldn't have enough hours.

Sorry.

[ikawrakow]

With PR #42 we get this

model	size	params	backend	threads	fa	test	t/s
IQ2_BN - 2.06 bpw TriLM	45.89 MiB	99.76 M	CPU	16	1	pp1500	12906.95 ± 61.04
IQ2_BN - 2.06 bpw TriLM	45.89 MiB	99.76 M	CPU	16	1	tg512	1563.62 ± 12.55

I.e., 56% improvement for PP and 26% improvement for TG since the original post from Aug 13!

I see PR-8151, which provides dedicated quantization for the TriLM ternary models in mainline llama.cpp, has been merged. Here is what we get for TQ2_0 that corresponds to our IQ2_TN

model	size	params	backend	threads	fa	test	t/s
TQ2_0 - 2.06 bpw ternary	45.89 MiB	99.76 M	CPU	16	1	pp1500	5187.34 ± 11.69
TQ2_0 - 2.06 bpw ternary	45.89 MiB	99.76 M	CPU	16	0	pp1500	5281.54 ± 53.33
TQ2_0 - 2.06 bpw ternary	45.89 MiB	99.76 M	CPU	16	1	tg500	1156.25 ± 18.14
TQ2_0 - 2.06 bpw ternary	45.89 MiB	99.76 M	CPU	16	0	tg500	1041.27 ± 21.30

Our version is 2.44X faster for PP and 35% faster for TG.