OpenVINO supports CPU, (i)GPU and NPU-enabled chips for Linux, Windows and macOS machines.
Could you please consider, if it's possible to add this library as a backend in Lc0 in order to run stuff on the NPU and GPU?
Comparing OpenVINO to Vulkan with llama.cpp and LLM Llama-3.2-1B-Instruct-f16.gguf shows OpenVINO@NPU equal to Vulkan@GPU, but with less power consumption. OpenVINO@GPU is more than twice as fast here.
Tested on Ubuntu 26.04 and Intel Core 258V laptop.
Thanks
OpenVINO NPU, GPU and CPU results
export GGML_OPENVINO_DEVICE=NPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
OpenVINO: using device NPU
| model |
size |
params |
backend |
ngl |
fa |
test |
t/s |
| llama 1B F16 |
2.30 GiB |
1.24 B |
OPENVINO |
99 |
1 |
pp512 |
2093.72 ± 18.93 |
| llama 1B F16 |
2.30 GiB |
1.24 B |
OPENVINO |
99 |
1 |
tg128 |
30.01 ± 0.11 |
export GGML_OPENVINO_DEVICE=GPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
OpenVINO: using device GPU
| model |
size |
params |
backend |
ngl |
fa |
test |
t/s |
| llama 1B F16 |
2.30 GiB |
1.24 B |
OPENVINO |
99 |
1 |
pp512 |
5748.75 ± 49.65 |
| llama 1B F16 |
2.30 GiB |
1.24 B |
OPENVINO |
99 |
1 |
tg128 |
27.33 ± 0.05 |
export GGML_OPENVINO_DEVICE=CPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1 --threads 4
OpenVINO: using device CPU
| model |
size |
params |
backend |
ngl |
fa |
test |
t/s |
| llama 1B F16 |
2.30 GiB |
1.24 B |
OPENVINO |
99 |
1 |
pp512 |
210.14 ± 1.97 |
| llama 1B F16 |
2.30 GiB |
1.24 B |
OPENVINO |
99 |
1 |
tg128 |
12.92 ± 1.66 |
Vulkan GPU result
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
load_backend: loaded RPC backend from ~/llama-b9553/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (LNL) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from ~/llama-b9553/libggml-vulkan.so
load_backend: loaded CPU backend from ~/llama-b9553/libggml-cpu-alderlake.so
| model |
size |
params |
backend |
ngl |
fa |
test |
t/s |
| llama 1B F16 |
2.30 GiB |
1.24 B |
Vulkan |
-1 |
1 |
pp512 |
2073.07 ± 39.28 |
| llama 1B F16 |
2.30 GiB |
1.24 B |
Vulkan |
-1 |
1 |
tg128 |
38.80 ± 0.27 |
OpenVINO supports CPU, (i)GPU and NPU-enabled chips for Linux, Windows and macOS machines.
Could you please consider, if it's possible to add this library as a backend in Lc0 in order to run stuff on the NPU and GPU?
Comparing
OpenVINOtoVulkanwith llama.cpp and LLM Llama-3.2-1B-Instruct-f16.gguf shows OpenVINO@NPU equal to Vulkan@GPU, but with less power consumption. OpenVINO@GPU is more than twice as fast here.Tested on Ubuntu 26.04 and Intel Core 258V laptop.
Thanks
OpenVINO NPU, GPU and CPU results
export GGML_OPENVINO_DEVICE=NPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
OpenVINO: using device NPUexport GGML_OPENVINO_DEVICE=GPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
OpenVINO: using device GPUexport GGML_OPENVINO_DEVICE=CPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1 --threads 4
OpenVINO: using device CPUVulkan GPU result
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
load_backend: loaded RPC backend from ~/llama-b9553/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (LNL) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from ~/llama-b9553/libggml-vulkan.so
load_backend: loaded CPU backend from ~/llama-b9553/libggml-cpu-alderlake.so