Cherry pick 12447#9

Merged

arthw merged 144 commits intomasterfrom

cherry-pick-12447

Mar 19, 2025

Commits on Mar 18, 2025

metal : copy kernels for quant to F32/F16 conversions (#12017)

authored and
arthw
committed
llama : expose llama_model_n_head_kv in the API (#11997)

vlovich
authored and
arthw
committed
Add Doc for Converting Granite Vision -> GGUF (#12006)

alex-jw-brooks
authored and
arthw
committed
server: support add_generation_prompt query param (#12062)

ochafik
authored and
arthw
committed
vulkan: implement more backpropagation operators (#11914)

remyoudompheng
authored and
arthw
committed
ggml-cpu: Fix build with sve (#12059)

MollySophia
authored and
arthw
committed
add OP sigmoid (#12056)

authored and
arthw
committed
server: handle echo=false on /v1/completions (#12060)

rhjdvsgsgks
authored and
arthw
committed
vulkan: fix assertion when qy_needs_dequant (#12068)

jeffbolznv
authored and
arthw
committed
docs: add docs/function-calling.md to lighten server/README.md's plight (#12069)

ochafik
authored and
arthw
committed
readme : update infra list (#9096)

kerthcet
authored and
arthw
committed
gguf-py: enable reading non-native endian files (#12081)

AlekseiNikiforovIBM
authored and
arthw
committed
Refactor gguf scripts to improve metadata handling (#11909)

CISC
authored and
arthw
committed
llava : add struct for FFI bindgen (#12079)

authored and
arthw
committed
cmake: Fix ggml backend dependencies and installation (#11818)

vvuksanovic
authored and
arthw
committed
vulkan: improve im2col (#11826)

daniandtheweb
authored and
arthw
committed
vulkan: matmul dequantization improvements (#12015)

netrunnereve
authored and
arthw
committed
CANN: Fix build error with GCC 13 (#11990)

hipudding
authored and
arthw
committed
ggml: aarch64: implement SVE kernels for q2_k_q8_k vector dot (#12064)

authored and
arthw
committed
CUDA: fix logic for V100 + GGML_CUDA_FORCE_MMQ (#12098)

JohannesGaessler
authored and
arthw
committed
vulkan: add specific MMV kernels for IQ2 and IQ3 quants + optimizations (#11595)

remyoudompheng
authored and
arthw
committed
Update granite vision docs for 3.2 model (#12105)

alex-jw-brooks
authored and
arthw
committed
llama : add Phi-4-mini support (supersede #12099) (#12108)

authored and
arthw
committed
ggml : upgrade init_tensor API to return a ggml_status (#11854)

authored and
arthw
committed
convert : fix Norway problem when parsing YAML (#12114)

ngxson
authored and
arthw
committed
webui : minor typo fixes (#12116)

authored and
arthw
committed
CUDA: compress mode option and default to size (#12029)

Green-Sky
authored and
arthw
committed
common : add --system-prompt parameter, replace behavior of -p in conversation mode (#12131)

authored and
arthw
committed
main: update outdated system prompt message (followup to #12131) (#12132)

authored and
arthw
committed
main: use jinja chat template system prompt by default (#12118)

authored and
arthw
committed
ggml-backend : keep paths in native string type when possible (#12144)

slaren
authored and
arthw
committed
SYCL: Move CPY kernels to a separate file and add few missing kernels (#12133)

qnixsynapse
authored and
arthw
committed
webui : add ?m=... and ?q=... params (#12148)

ngxson
authored and
arthw
committed
Adding UTF-8 support to llama.cpp (#12111)

ericcurtin
authored and
arthw
committed
ggml : fix kleidiai build (#12159)

ag2s20150909
authored and
arthw
committed
test-backend-ops : add option -p to filter by op params (#12155)

slaren
authored and
arthw
committed
tts: add speaker file support (#12048)

authored and
arthw
committed
ci : set GITHUB_ACTION env var for server tests (#12162)

danbev
authored and
arthw
committed
scripts : sync-ggml-am.sh fix

ggerganov
authored and
arthw
committed
Support pure float16 add/sub/mul/div operations in the CUDA (and CPU) backend (ggml/1121)

cmdr2
authored and
arthw
committed
Told cmake to install ggml-cpp.h as a public header file. (ggml/1126)

authored and
arthw
committed
cmake : fix compile assumptions for power9/etc (whisper/2777)

authored and
arthw
committed
whisper : support GGML_BACKEND_DL (whisper/2843)

authored and
arthw
committed
cuda/cpu: Increase support for fp16 unary operations (ggml/1125)

cmdr2
authored and
arthw
committed
sync : ggml

ggerganov
authored and
arthw
committed
cuda/vulkan: specify fp32-only support for some operations in supports_op (ggml/1129)

cmdr2
authored and
arthw
committed
sync : ggml

ggerganov
authored and
arthw
committed
cuda: unary ops as float + de-duplicate (ggml/1130)

cmdr2
authored and
arthw
committed
sync : ggml

ggerganov
authored and
arthw
committed
HIP: implement FlashAttention via rocWMMA for CDNA and RDNA3+ (#12032)

authored and
arthw
committed
`server`: fix deadly typo in response_format.json_schema.schema handling (#12168)

ochafik
authored and
arthw
committed
main: allow preloading conversation with -p and add -st / --single-turn (#12145)

CISC
authored and
arthw
committed
readme : fix roadmap link (#12185)

ggerganov
authored and
arthw
committed
ggml : portability fixes for VS 2017 (#12150)

authored and
arthw
committed
llama : add xcframework build script (#11996)

danbev
authored and
arthw
committed
server : fix cache reuse logic (#12161)

Clauszy
authored and
arthw
committed
ci : remove xframework upload (#12190)

danbev
authored and
arthw
committed
ci : fix xcframework artifact tag (#12191)

danbev
authored and
arthw
committed
`tool-call`: fix Qwen 2.5 Coder support, add micro benchmarks, support trigger patterns for lazy grammars (#12034)

authored and
arthw
committed
ci : add fetch-depth to xcframework upload (#12195)

danbev
authored and
arthw
committed
ggml : fix GGMLMetalClass ODR (#12200)

pminev
authored and
arthw
committed
ggml-cpu: Faster IQ1 mul_mat_vec on AVX2 using BMI2 instructions (#12154)

remyoudompheng
authored and
arthw
committed
opencl : fix profile-related errors (#12095)

authored and
arthw
committed
opencl : fix `ulong` kernel args were set from `int` variables (#12174)

linehill
authored and
arthw
committed
opencl : fix buffer alignment (#12197)

linehill
authored and
arthw
committed
android : fix KV cache log message condition (#12212)

hanyin-arm
authored and
arthw
committed
HIP/CUDA: set the paramerter value in maintain_cuda_graph instead of replaceing it. (#12209)

IMbackK
authored and
arthw
committed
llava: add big-endian conversion for image encoder (#12218)

taronaeo
authored and
arthw
committed
update function-calling.md w/ template override for functionary-small-v3.2 (#12214)

ochafik
authored and
arthw
committed
HIP: rocWMMA documentation and enabling in workflow builds (#12179)

hjc4869
authored and
arthw
committed
CUDA: fix FA logic for PTX 7.0 and CC >= 7.5 (#12222)

JohannesGaessler
authored and
arthw
committed
readme : update bindings (#12229)

lmbelo
authored and
arthw
committed
cmake : fix undefined reference errors for std::filesystem in ggml (#12092) (#12094)

authored and
arthw
committed
opencl: Noncontiguous `norm`, `rms_norm`, disable `fp16` for some ops (#12217)

lhez
authored and
arthw
committed
metal : fix default.metallib build (#12224)

danbev
authored and
arthw
committed
HIP: fix rocWMMA build flags under Windows (#12230)

hjc4869
authored and
arthw
committed
metal : simplify kernel arguments using a struct (#3229) (#12194)

authored and
arthw
committed
sync: minja - support QwQ-32B (#12235)

ochafik
authored and
arthw
committed
server : Log original chat template parsing error (#12233)

CISC
authored and
arthw
committed
ci : fix save-load test invocations (#12245)

ggerganov
authored and
arthw
committed
ggml-cpu: faster AVX2 variant for IQ1_M (#12216)

remyoudompheng
authored and
arthw
committed
ggml : ggml_compute_forward_concat() for arbitrary tensor type (ggml/1118)

vmobilis
authored and
arthw
committed
sync : ggml

ggerganov
authored and
arthw
committed
ggml : skip intermediate .air file when compiling .metallib (#12247)

danbev
authored and
arthw
committed
server : infill gen ends on new line (#12254)

ggerganov
authored and
arthw
committed
ggml-backend : make path_str compatible with C++20 (#12269)

ctrysbita
authored and
arthw
committed
authors : update (#12271)

ggerganov
authored and
arthw
committed
server : add speculative decoding presets for FIM (#12287)

ggerganov
authored and
arthw
committed
llava : fix bug in minicpm-v code (#11513)

tc-mb
authored and
arthw
committed
`sampler`: fixes trigger tokens + lazy grammars (fix typo cast from token to string) (#12291)

authored and
arthw
committed
allow missing content in message if tool_calls provided (#12293)

ochafik
authored and
arthw
committed
`tool-call`: ensure there's always a non-empty tool call id (#12292)

ochafik
authored and
arthw
committed
`server`: extract <think> tags from qwq outputs (#12297)

ochafik
authored and
arthw
committed
common : refactor '-o' option (#12278)

marcoStocchi
authored and
arthw
committed
tests : fix test-quantize-fns to init the CPU backend (#12306)

ggerganov
authored and
arthw
committed
readme: added Sidekick to available UIs (#12311)

johnbean393
authored and
arthw
committed
opencl: use OpenCL C standard supported by the device (#12221)

Henry Linjamäki
authored and
arthw
committed
musa: support new arch mp_31 and update doc (#12296)

yeahdongcn
authored and
arthw
committed
mat vec double buffer (#12188)

netrunnereve
authored and
arthw
committed
clip : bring back GPU support (#12322)

ngxson
authored and
arthw
committed
metal : Cache the Metal library at the device context level (#12265)

BB-fat
authored and
arthw
committed
ggml-backend : fix backend search path (#12330)

jklincn
authored and
arthw
committed
CUDA/HIP: refractor mmqv to unify the calculation of nwarps and rows per block between host and device code. (#12177)

authored and
arthw
committed
vulkan: fix bug in coopmat1 mul_mat_id (#12316)

jeffbolznv
authored and
arthw
committed
llama : Add Gemma 3 support (+ experimental vision capability) (#12343)

ngxson
authored and
arthw
committed
CUDA/HIP: Fix fattn-vec-* when device warp size is not 32 (#12315)

IMbackK
authored and
arthw
committed

Commits on Mar 19, 2025

llama.swiftui : fix xcframework dir in README [no ci] (#12353)

danbev
authored and
arthw
committed
Update build.yml for Windows Vulkan builder to use Vulkan 1.4.304 SDK for VK_NV_cooperative_matrix2 support (#12301)

oscarbg
authored and
arthw
committed
server : fix crash when using verbose output with input tokens that are not in printable range (#12178) (#12338)

authored and
arthw
committed
llama : refactor llama_context, llama_kv_cache, llm_build_context (#12181)

ggerganov
authored and
arthw
committed
arg : no n_predict = -2 for examples except for main and infill (#12364)

ngxson
authored and
arthw
committed
llama : fix Gemma3 SWA KV cache shift (#12373)

ggerganov
authored and
arthw
committed
hparams : add SWA rope parameters (#12374)

ggerganov
authored and
arthw
committed
graph : simplify attn input build for unified KV cache (#12381)

ggerganov
authored and
arthw
committed
server: fix "--grammar-file" parameter (#12285)

dodekapod
authored and
arthw
committed
Load all MoE experts during warmup (#11571)

authored and
arthw
committed
main : add -sysf / --system-prompt-file (#12249) (#12250)

CISC
authored and
arthw
committed
Add CLI arg to llama-run to adjust the number of threads used (#12370)

ericcurtin
authored and
arthw
committed
[CANN]MUL_MAT optimization (#12382)

noemotiovon
authored and
arthw
committed
SYCL : support non-contiguous tensors in binary ops (add, sub, etc) (#12399)

authored and
arthw
committed
SYCL: Delete redundant plus sign and space (#12391)

aubreyli
authored and
arthw
committed
llama-tts : add '-o' option (#12398)

marcoStocchi
authored and
arthw
committed
ci : add --symlinks to xcframework zip command (#12409)

danbev
authored and
arthw
committed
context : fix init of n_outputs (#12397)

ggerganov
authored and
arthw
committed
llama : fix OLMo-2-0325-32B-Instruct K-norm size (#12400)

CISC
authored and
arthw
committed
SYCL: set extras only on GGML_TYPE_Q4_0 (#12366)

qnixsynapse
authored and
arthw
committed
cmake : enable building llama.cpp using system libggml (#12321)

ckastner
authored and
arthw
committed
vulkan: Adjust coopmat2 tile sizes and selection heuristic (#12258)

jeffbolznv
authored and
arthw
committed
vulkan: Pad N dimension of B matrix for coopmat2 perf, to avoid bounds checking (#12273)

jeffbolznv
authored and
arthw
committed
vulkan: use fp32 in coopmat2 q4_k dequant function (#12309)

jeffbolznv
authored and
arthw
committed
vulkan: subgroup size tuning (#12087)

authored and
arthw
committed
vulkan: Add N/2 and N/4 optimized paths in coopmat2 shader (#12312)

jeffbolznv
authored and
arthw
committed
ggml-vulkan: remove unused find_program(glslc) (#12416)

guusw
authored and
arthw
committed
cuda : enable CUDA Graph on CUDA Toolkit < 12.x (#12394)

gaugarg-nv
authored and
arthw
committed
docs : bring llama-cli conversation/template docs up-to-date (#12426)

CISC
authored and
arthw
committed
llama: Add support for RWKV v7 architecture (#12412)

MollySophia
authored and
arthw
committed
fixed compilation warnings in ggml-sycl (#12424)

lslusarczyk
authored and
arthw
committed
Vulkan: Default to 1GB allocations instead of 4GB to avoid fragmentation and driver issues (#12434)

0cc4m
authored and
arthw
committed
ggml : add SVE support for q6_K_q8_K (#12361)

fj-y-saito
authored and
arthw
committed
cmake : fix PowerPC build (#12241)

mehendarkarprajwal
authored and
arthw
committed
server : fix warmup draft cache type (#12446)

ggerganov
authored and
arthw
committed
context : always use non-causal attention for encoder graphs (#12447)

ggerganov
authored and
arthw
committed
rm other backend ci, keep cpu, rpc, sycl
arthw
committed
rm tail space
arthw
committed