Misc. bug: Metal running out of memory after PR #15966

[kunle12]

Name and Version

% build/bin/llama-cli --version
version: 6473 (9dcd200)
built with Apple clang version 17.0.0 (clang-1700.0.13.5) for arm64-apple-darwin24.6.0

Operating systems

Mac

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./llama-server -m ~/qwen3/Qwen_Qwen3-235B-A22B-Instruct-2507-IQ3_XS-00001-of-00003.gguf --temp 0.6 --no-webui -t 6 --host 0.0.0.0 --port 8081

Problem description & steps to reproduce

System configuration: Mac Studio M4 Max 128GB unified memory. OS: 15.7.1

I was able to run Qwen3-235B-A22B-Instruct-2507 IQ3 quantised model on the machine without issue prior to PR #15966.

After the removal of memory pool, I am hitting out of memory error

ggml_backend_metal_synchronize: error: command buffer 0 failed with status 5 error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Not sure this could be considered as a bug. Perhaps some workaround will be nice.

First Bad Commit

commit 9dcd200 (HEAD, tag: b6473)

Relevant log output

[ggerganov]

How big is this model on disk?

I think it's normal that after this change, the memory usage increased a bit. Most likely you were just at the limit of your machine before that and now you crossed it. You can increase it with:

sudo sysctl iogpu.wired_limit_mb=120000

[kunle12]

iogpu.wired_limit_mb

Yep. You are correct, increasing the limit does resolve the issue. Thanks!

[ggerganov]

Indeed there is an unexpected effect here causing more memory usage than necessary that I didn't notice when making this change. I think that the allocator logic is somehow failing to reuse old memory blocks when the graph has mul_mat_id nodes, but haven't been able to fully trace the root cause yet.

For example, on master, this command allocates a very large compute buffer:

./bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja -ub 2048 -c 131072

llama_context: Flash Attention was auto, set to enabled
llama_context:      Metal compute buffer size =  8264.84 MiB
llama_context:        CPU compute buffer size =  1064.59 MiB
llama_context: graph nodes  = 1352
llama_context: graph splits = 2

In contrast, if I remove the small fleeting buffers for MUL_MAT_ID, the compute buffer drops to expected size:

diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal.cpp
index 7afc881fa..5c947811b 100644
--- a/ggml/src/ggml-metal/ggml-metal.cpp
+++ b/ggml/src/ggml-metal/ggml-metal.cpp
@@ -188,8 +188,8 @@ static size_t ggml_backend_metal_buffer_type_get_alloc_size(ggml_backend_buffer_
     switch (tensor->op) {
         case GGML_OP_MUL_MAT_ID:
             {
-                res += ggml_metal_op_mul_mat_id_extra_tpe(tensor);
-                res += ggml_metal_op_mul_mat_id_extra_ids(tensor);
+                //res += ggml_metal_op_mul_mat_id_extra_tpe(tensor);
+                //res += ggml_metal_op_mul_mat_id_extra_ids(tensor);
             } break;
         case GGML_OP_FLASH_ATTN_EXT:
             {

llama_context: Flash Attention was auto, set to enabled
llama_context:      Metal compute buffer size =  1672.08 MiB
llama_context:        CPU compute buffer size =  1064.59 MiB
llama_context: graph nodes  = 1352
llama_context: graph splits = 2

Alternatively, if I superficially expand also the ADD_ID buffers to match the size of the MUL_MAT_ID buffers also helps, but not completely:

diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal.cpp
index 7afc881fa..a7dbf3234 100644
--- a/ggml/src/ggml-metal/ggml-metal.cpp
+++ b/ggml/src/ggml-metal/ggml-metal.cpp
@@ -187,6 +187,7 @@ static size_t ggml_backend_metal_buffer_type_get_alloc_size(ggml_backend_buffer_
     // some operations require additional memory for fleeting data:
     switch (tensor->op) {
         case GGML_OP_MUL_MAT_ID:
+        case GGML_OP_ADD_ID:
             {
                 res += ggml_metal_op_mul_mat_id_extra_tpe(tensor);
                 res += ggml_metal_op_mul_mat_id_extra_ids(tensor);

llama_context: Flash Attention was auto, set to enabled
llama_context:      Metal compute buffer size =  4294.09 MiB
llama_context:        CPU compute buffer size =  1064.59 MiB
llama_context: graph nodes  = 1352
llama_context: graph splits = 2

Debugging the allocator, I observe that when we create fleeting buffers for the MUL_MAT_ID the number of free blocks in the chunk increase with each layer, as opposed to no creating those. So this makes me think that the logic for reusing blocks fails in this case, but not sure why yet.

cc @slaren in case you have ideas

[ggerganov]

Fixed with #16679