Why larger ubatch size didn't improve performance when inference with cpu on Android devices?

[Msyu1020]

Hi, I’m trying to understand how ubatch sizes impact inference performance on Android devices. I’ve verified that when using the GPU for inference, different ubatch sizes result in varying performance on my Xiaomi 14 Pro. However, when using the CPU, the ubatch size doesn’t seem to affect performance significantly.

In my view, a larger ubatch size allows for the calculation of more tokens at once, but it also requires a larger memory allocation to store temporary variables. I'm curious why the performance on the CPU shows almost no difference with different ubatch sizes when all other settings remain the same.

• Code version: b5811
• Device: Xiaomi 14 Pro ( Android / Adreno GPU )

CPU

model	size	params	backend	ngl	n_ubatch	test	pp t/s
llama 7B Q4_0	3.56 GiB	6.74 B	OpenCL	0	64	pp64	8.47
llama 7B Q4_0	3.56 GiB	6.74 B	OpenCL	0	32	pp64	8.76
llama 7B Q4_0	3.56 GiB	6.74 B	OpenCL	0	16	pp64	8.65

GPU

model	size	params	backend	ngl	n_ubatch	test	pp t/s
llama 7B Q4_0	3.56 GiB	6.74 B	OpenCL	99	64	pp64	73.35
llama 7B Q4_0	3.56 GiB	6.74 B	OpenCL	99	32	pp64	54.17
llama 7B Q4_0	3.56 GiB	6.74 B	OpenCL	99	16	pp64	37.52