Skip to content

[Feature request] Add OpenVINO backend #2419

Description

@twoplan

OpenVINO supports CPU, (i)GPU and NPU-enabled chips for Linux, Windows and macOS machines.

Could you please consider, if it's possible to add this library as a backend in Lc0 in order to run stuff on the NPU and GPU?

Comparing OpenVINO to Vulkan with llama.cpp and LLM Llama-3.2-1B-Instruct-f16.gguf shows OpenVINO@NPU equal to Vulkan@GPU, but with less power consumption. OpenVINO@GPU is more than twice as fast here.
Tested on Ubuntu 26.04 and Intel Core 258V laptop.

Thanks

OpenVINO NPU, GPU and CPU results
export GGML_OPENVINO_DEVICE=NPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
OpenVINO: using device NPU

model size params backend ngl fa test t/s
llama 1B F16 2.30 GiB 1.24 B OPENVINO 99 1 pp512 2093.72 ± 18.93
llama 1B F16 2.30 GiB 1.24 B OPENVINO 99 1 tg128 30.01 ± 0.11

export GGML_OPENVINO_DEVICE=GPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
OpenVINO: using device GPU

model size params backend ngl fa test t/s
llama 1B F16 2.30 GiB 1.24 B OPENVINO 99 1 pp512 5748.75 ± 49.65
llama 1B F16 2.30 GiB 1.24 B OPENVINO 99 1 tg128 27.33 ± 0.05

export GGML_OPENVINO_DEVICE=CPU
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1 --threads 4
OpenVINO: using device CPU

model size params backend ngl fa test t/s
llama 1B F16 2.30 GiB 1.24 B OPENVINO 99 1 pp512 210.14 ± 1.97
llama 1B F16 2.30 GiB 1.24 B OPENVINO 99 1 tg128 12.92 ± 1.66

Vulkan GPU result
./llama-bench -m ~/models/Llama-3.2-1B-Instruct-f16.gguf -fa 1
load_backend: loaded RPC backend from ~/llama-b9553/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (LNL) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from ~/llama-b9553/libggml-vulkan.so
load_backend: loaded CPU backend from ~/llama-b9553/libggml-cpu-alderlake.so

model size params backend ngl fa test t/s
llama 1B F16 2.30 GiB 1.24 B Vulkan -1 1 pp512 2073.07 ± 39.28
llama 1B F16 2.30 GiB 1.24 B Vulkan -1 1 tg128 38.80 ± 0.27

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions