NPU support for LLM acceleration without gpu

[sebastienbo]

Is it possible to support NPU's ? Those are way more specialised for LLM traversal and don't require a gpu.
It would offload the cpu and would be able to run from Laptops with NPU's
Here is an example code: https://intel.github.io/intel-npu-acceleration-library/llm.html

[linzj]

I concur with your perspective; acquiring a 64GB DDR5 RAM module is indeed more feasible compared to obtaining a 64GB GPU at present.Indeed, incorporating NPU support holds the promise of delivering significant advantages to users in terms of model inference compared to solely relying on GPU support.

[cebtenzzre]

If we did add support for NPUs the code would go in llama.cpp, which we use for LLM inference. See these upstream issues:

• [Feature request] Any plans for AMD XDNA AI Engine support on Ryzen 7x40 processors? ggml-org/llama.cpp#1499
• will this project support on device npu like qualcomm hexagon？ ggml-org/llama.cpp#2687