SDPA f32#15064

Closed

wine99 wants to merge 110 commits intoggml-org:masterfrom

Commits on Jul 30, 2025

Add ggml-openvino base files

YangleiZouIntel
authored and
ravi9
committed
add openvino as optional backend for Llama.cpp ggml

zhanmyz
authored and
ravi9
committed
* Configure the device(default CPU) that uses OpenVINO to compile the model

zhanmyz
authored and
ravi9
committed
Solve the issue of abnormal model output caused by using OpenVINO ADD operator

zhanmyz
authored and
ravi9
committed
Add OpenVINO MUL operator to GGML of Llama.cpp.

zhanmyz
authored and
ravi9
committed
Add compile options

zhanmyz
authored and
ravi9
committed
add OpenVINO frontend convert process steps

zhanmyz
authored and
ravi9
committed
add get openvino available ops function

zhanmyz
authored and
ravi9
committed
Add PoC of integration of openvino frontend. Main changes: ggml-ov-frontend-utils, GraphIterator, Decoder

yumengbo
authored and
ravi9
committed
Implement GgmlOvDecoder. Add dump functions.

yumengbo
authored and
ravi9
committed
Convert subgraph with add, sub, mul, div op to ov model and do infer on openvino device

yumengbo
authored and
ravi9
committed
Add GGML_OV_FRONTEND option. Add readme.

yumengbo
authored and
ravi9
committed
Change output for infer request to set output tensor. Support scale, view op.

yumengbo
authored and
ravi9
committed
add GET_ROWS operator of OpenVINO to GGML of llama.cpp

zhanmyz
authored and
ravi9
committed
Update build.md and add operation mapping(GGML to OpenVINO)

zhanmyz
authored and
ravi9
committed
add the rms_norm operator implemented using OpenVINO to the GGML backend of llama.cpp

zhanmyz
authored and
ravi9
committed
Fix issue for output memory copy of infer request

yumengbo
authored and
ravi9
committed
Change to implementation following pytorch frontend

yumengbo
authored and
ravi9
committed
Add support for UNARY SILU op . Fix pytorch impl bugs.

yumengbo
authored and
ravi9
committed
Support Softmax op

yumengbo
authored and
ravi9
committed
Support Softmax op

yumengbo
authored and
ravi9
committed
Support ROPE op.

yumengbo
authored and
ravi9
committed
Add support for RMS_NORM OP

zhanmyz
authored and
ravi9
committed
Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML backend

zhanmyz
authored and
ravi9
committed
Move CPY from GGML OV Backend to OV Frontend

zhanmyz
authored and
ravi9
committed
add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops

zhanmyz
authored and
ravi9
committed
add implementation of CPY when the output tensor is non-contiguous

zhanmyz
authored and
ravi9
committed
add tmp source code files

zhanmyz
authored and
ravi9
committed
Execute singel CONT operator is OK

zhanmyz
authored and
ravi9
committed
Execute CONT & VIEW operators in OV Frontend is OK

zhanmyz
authored and
ravi9
committed
OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT graph conversion of consecutive OPs

zhanmyz
authored and
ravi9
committed
OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT/ROPE/SCALE/SOFTMAX/ADD adjacent op graph conversion

zhanmyz
authored and
ravi9
committed
Change the input parameter shape of CONT operator

zhanmyz
authored and
ravi9
committed
Change the input and ouput node shape of MUL_MAT operator

zhanmyz
authored and
ravi9
committed
Change the input and ouput node shape of MUL_MAT operator

zhanmyz
authored and
ravi9
committed
change CONT and MULMAT input node shape

zhanmyz
authored and
ravi9
committed
All adjacent ops can conversion but calculation result is wrong and need debugging

zhanmyz
authored and
ravi9
committed
1. All operators implemented using OpenVINO can be successfully executed individually.

zhanmyz
authored and
ravi9
committed
1. Update the implementation of CPY node when it's non-contiguous

zhanmyz
authored and
ravi9
committed
Minor Update

zhanmyz
authored and
ravi9
committed
Try to add VIEW node to OV Frontend and have some issues that need to be dealt with

zhanmyz
authored and
ravi9
committed
1. In the Prompt process and predict first token stage, the PERMUTE node needs to be integrated into the OV Frontend

zhanmyz
authored and
ravi9
committed
add debug info

zhanmyz
authored and
ravi9
committed
Process Prompt and predict first token is OK

zhanmyz
authored and
ravi9
committed
1. Solve the AC issue of Permute+VIEW and MULMAL issue in the phase of “1. Process Prompt and predict the first token”.

zhanmyz
authored and
ravi9
committed
1. Delete some comments

zhanmyz
authored and
ravi9
committed
* Use find_package in CMake to configure OpenVINO

wine99
authored and
ravi9
committed
change op mappings to list in openvino_supports_op

wine99
authored and
ravi9
committed
2nd+ token correct by fix CPY in OV, remove single op backend compute code

wine99
authored and
ravi9
committed
Arbitrary token len (>32) work; Fix bug in mulmat

wine99
authored and
ravi9
committed
FEAT: do PERMUTE eagerly

wine99
authored and
ravi9
committed
FEAT: Add interleaved mode for ROPE

wine99
authored and
ravi9
committed
REFACTOR: support weigts as constant

wine99
authored and
ravi9
committed
STYLE: minor refactor

wine99
authored and
ravi9
committed
PERF: share const nodes for weights for diff infer

wine99
authored and
ravi9
committed
BUILD: update build doc, add cmake preset, add CACHE_DIR env var

wine99
authored and
ravi9
committed
FEAT: improve debug capability

wine99
authored and
ravi9
committed
PERF: compile once (dynamic graph + cache)

wine99
authored and
ravi9
committed
Rebase - Bring up to date and fix build process

virajwad
authored and
ravi9
committed
fix build error

wine99
authored and
ravi9
committed
FIX: backend buffer type issue

wine99
authored and
ravi9
committed
STYLE: clang-format

wine99
authored and
ravi9
committed
FEAT: Add all conversion code from ov side

wine99
authored and
ravi9
committed
PERF: favor low precision matmul

wine99
authored and
ravi9
committed
STYLE and minor REFACTOR

wine99
authored and
ravi9
committed
FIX: Re-add tensor names in cgraph, Add another case for RESHAPE

wine99
authored and
ravi9
committed
FIX: input shape of KQ_mask

wine99
authored and
ravi9
committed
PERF: add weight constant in parallel

wine99
authored and
ravi9
committed
FIX: set_max_token_len

wine99
authored and
ravi9
committed
PERF: use Slice+Concat in writing cache_v

wine99
authored and
ravi9
committed
Update build doc

wine99
authored and
ravi9
committed
Add cgraph tensor output name to OV op name

wine99
authored and
ravi9
committed
Update openvino build instructions
ravi9
committed
Add initial NPU support

wine99
authored and
ravi9
committed
draft NPU support version 2: prefill + kvcache

wine99
authored and
ravi9
committed
NPU support version 2: prefill + kvcache

wine99
authored and
ravi9
committed
Change due to ggml cgraph changes, not correct yet

wine99
authored and
ravi9
committed
Change due to ggml cgraph changes, llama-3.2 CPU work

wine99
authored and
ravi9
committed
Add AMD64 to CMakeLists

wine99
authored and
ravi9
committed
Change due to ggml cgraph changes, all device work

wine99
authored and
ravi9
committed
Refactor: clean, fix warning

wine99
authored and
ravi9
committed
Update clang-format

wine99
authored and
ravi9
committed
Statful transformation for CPU GPU

wine99
authored and
ravi9
committed
Add SwiGLU

wine99
authored and
ravi9
committed
Fuse to SDPA

wine99
authored and
ravi9
committed
Replace Concat with Broadcast in MulMat for GQA

wine99
authored and
ravi9
committed
Pull out indices creation for kv cache update

wine99
authored and
ravi9
committed
Refactor: remove past_token_len from extra_inputs

wine99
authored and
ravi9
committed
Fix Phi3 SwiGLU and SoftMax

wine99
authored and
ravi9
committed
Pull out sin cos from rope

wine99
authored and
ravi9
committed
Reduce memory: free ov weights node after graph conversion

wine99
authored and
ravi9
committed
Fix CPY due to cgraph change

wine99
authored and
ravi9
committed
Added OpenVINO CI/CD. Updated docs
ravi9
committed
Fix llama-cli

wine99
authored and
ravi9
committed
Fix Phi3 ROPE; Add test-backend-ops

wine99
authored and
ravi9
committed
Fix NPU

wine99
authored and
ravi9
committed
Fix llama-bench; Clang-format

wine99
authored and
ravi9
committed
Fix llama-perplexity

wine99
authored and
ravi9
committed
temp. changes for mark decomp

cavusmustafa
authored and
ravi9
committed
matmul in fp32

wine99
authored and
ravi9
committed
mulmat input conversion fix

cavusmustafa
authored and
ravi9
committed
mulmat type conversion update

cavusmustafa
authored and
ravi9
committed
add mark decomp pass

cavusmustafa
authored and
ravi9
committed
Revert changes in fuse_to_sdpa

wine99
authored and
ravi9
committed

Commits on Aug 1, 2025

Use CiD for NPU
wine99
committed

Commits on Aug 4, 2025

SDPA in f32
wine99
committed