Misc. bug: llama-fit-params stuck in infinite loop

[jettoblack]

Name and Version

version: 7524 (5ee4e43)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

Other (Please specify in the next section)

Command line

llama-fit-params -hf unsloth/GLM-4.7-GGUF:IQ2_XXS -c 65536

Problem description & steps to reproduce

On a host with 4x 3090 24GB, 1x 5060Ti 16GB, 64GB host RAM

Output without --verbose, here's where it gets stuck and stops logging:

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 5 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 4: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
common_download_file_single_online: using cached file: /home/jasonl/.cache/llama.cpp/unsloth_GLM-4.7-GGUF_UD-IQ2_XXS_GLM-4.7-UD-IQ2_XXS-00001-of-00003.gguf
common_download_file_single_online: using cached file: /home/jasonl/.cache/llama.cpp/unsloth_GLM-4.7-GGUF_UD-IQ2_XXS_GLM-4.7-UD-IQ2_XXS-00002-of-00003.gguf
common_download_file_single_online: using cached file: /home/jasonl/.cache/llama.cpp/unsloth_GLM-4.7-GGUF_UD-IQ2_XXS_GLM-4.7-UD-IQ2_XXS-00003-of-00003.gguf
build: 7524 (5ee4e43f2) with GNU 13.3.0 for Linux x86_64
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 3090)   :  24124 total,  27958 used,   4099 deficit
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 3090)   :  24124 total,  30479 used,   6620 deficit
llama_params_fit_impl:   - CUDA2 (NVIDIA GeForce RTX 3090)   :  24124 total,  29214 used,   5355 deficit
llama_params_fit_impl:   - CUDA3 (NVIDIA GeForce RTX 3090)   :  24115 total,  28974 used,   5131 deficit
llama_params_fit_impl:   - CUDA4 (NVIDIA GeForce RTX 5060 Ti):  15848 total,  18177 used,   2468 deficit
llama_params_fit_impl: projected to use 134804 MiB of device memory vs. 112335 MiB of free device memory
llama_params_fit_impl: cannot fulfill margin of 1024 MiB on all devices, need to use 28796 MiB less in total
llama_params_fit_impl: context size set by user to 65536 -> no change
llama_params_fit_impl: with only dense weights in device memory there is a total surplus of 71665 MiB
llama_params_fit_impl: filling dense-only layers back-to-front:
llama_params_fit_impl:   - CUDA4 (NVIDIA GeForce RTX 5060 Ti): 41 layers,  14446 MiB used,   1262 MiB free
llama_params_fit_impl:   - CUDA3 (NVIDIA GeForce RTX 3090)   : 53 layers,  18698 MiB used,   5143 MiB free
llama_params_fit_impl:   - CUDA2 (NVIDIA GeForce RTX 3090)   :  0 layers,      0 MiB used,  23859 MiB free
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 3090)   :  0 layers,      0 MiB used,  23859 MiB free
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 3090)   :  0 layers,   1032 MiB used,  22826 MiB free
llama_params_fit_impl: converting dense-only layers to full layers and filling them front-to-back with overflow to next device/system memory:
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 3090)   : 18 layers ( 1 overflowing),  22833 MiB used,   1025 MiB free
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 3090)   : 15 layers ( 1 overflowing),  22430 MiB used,   1428 MiB free
llama_params_fit_impl:   - CUDA2 (NVIDIA GeForce RTX 3090)   : 15 layers ( 1 overflowing),  22657 MiB used,   1201 MiB free

First Bad Commit

No response

Relevant log output

When running with --verbose, the output log grows seemingly forever, or at least as long as I was willing to let it run. Here's ~ 200MB worth zipped:

[fit.txt.gz](https://github.com/user-attachments/files/24321672/fit.txt.gz)

[JohannesGaessler]

I think there is a problem with uploading files in text blocks. When I copy-paste the link I get "Not Found". Can you upload the file again?

[jettoblack]

fit.txt.gz

Trying the attachment again.

I can reproduce the issue with other models and with certain context sizes, but changing the context size by a few K often fixes it, e.g. 65536 and 80000 will reproduce the problem, but 72000 will work.

[JohannesGaessler]

Should be fixed by #18415 , please confirm.

[jettoblack]

@JohannesGaessler Sorry to say after trying this fix I'm still experiencing the problem, using build 7565 (382808c) which includes PR #18415 .

Here's a verbose log of one that still causes an infinite loop:

build/bin/llama-server -hf unsloth/MiniMax-M2.1-GGUF:Q3_K_XL -c 131072 --host 0.0.0.0 --jinja --verbose

fit_7565.txt.gz

Here's where that command gets stuck without --verbose:

llama_params_fit_impl: converting dense-only layers to full layers and filling them front-to-back with overflow to next d
evice/system memory:
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 3090)   : 11 layers ( 1 overflowing),  22408 MiB used,   1450 MiB fr
ee
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 3090)   : 11 layers ( 1 overflowing),  22473 MiB used,   1385 MiB fr
ee
llama_params_fit_impl:   - CUDA2 (NVIDIA GeForce RTX 3090)   : 11 layers ( 1 overflowing),  22785 MiB used,   1073 MiB fr
ee

As I mentioned I can reproduce the problem with various models and context sizes.

Here's a different, possibly unrelated issue where changing the context size to 196608 (the max size for this model) avoids the infinite loop, but the resulting fit fails at load time, it is trying to load > 17GB onto the 16GB GPU:

build/bin/llama-server -hf unsloth/MiniMax-M2.1-GGUF:Q3_K_XL --host 0.0.0.0 --jinja -c 196608

llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 3090)   :  24124 total,  34256 used,  10397 deficit
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 3090)   :  24124 total,  33138 used,   9279 deficit
llama_params_fit_impl:   - CUDA2 (NVIDIA GeForce RTX 3090)   :  24124 total,  30700 used,   6841 deficit
llama_params_fit_impl:   - CUDA3 (NVIDIA GeForce RTX 3090)   :  24115 total,  33305 used,   9463 deficit
llama_params_fit_impl:   - CUDA4 (NVIDIA GeForce RTX 5060 Ti):  15848 total,  18283 used,   2574 deficit
llama_params_fit_impl: projected to use 149685 MiB of device memory vs. 111128 MiB of free device memory
llama_params_fit_impl: cannot fulfill margin of 1024 MiB on all devices, need to use 43677 MiB less in total
llama_params_fit_impl: context size set by user to 196608 -> no change
llama_params_fit_impl: with only dense weights in device memory there is a total surplus of 53844 MiB
llama_params_fit_impl: filling dense-only layers back-to-front:
llama_params_fit_impl:   - CUDA4 (NVIDIA GeForce RTX 5060 Ti): 18 layers,  14421 MiB used,   1287 MiB free
llama_params_fit_impl:   - CUDA3 (NVIDIA GeForce RTX 3090)   : 28 layers,  22489 MiB used,   1352 MiB free
llama_params_fit_impl:   - CUDA2 (NVIDIA GeForce RTX 3090)   : 17 layers,  14149 MiB used,   9709 MiB free
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 3090)   :  0 layers,      0 MiB used,  23859 MiB free
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 3090)   :  0 layers,   1149 MiB used,  22709 MiB free
llama_params_fit_impl: converting dense-only layers to full layers and filling them front-to-back with overflow to next device/system memory:
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 3090)   :  9 layers ( 0 overflowing),  22144 MiB used,   1714 MiB free
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 3090)   : 10 layers ( 1 overflowing),  22588 MiB used,   1270 MiB free
llama_params_fit_impl:   - CUDA2 (NVIDIA GeForce RTX 3090)   : 10 layers ( 1 overflowing),  22409 MiB used,   1449 MiB free
llama_params_fit_impl:   - CUDA3 (NVIDIA GeForce RTX 3090)   : 16 layers (11 overflowing),  21910 MiB used,   1931 MiB free
llama_params_fit: successfully fit params to free device memory
[...]
load_tensors: loading model tensors, this can take a while... (mmap = true)
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 17762.63 MiB on device 4: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA4 buffer of size 18625472512
llama_model_load: error loading model: unable to allocate CUDA4 buffer

Verbose log for that one:

fit_7565_toobig.txt.gz

Let me know if that should be written up as a separate issue.

[JohannesGaessler]

Sorry, I was too hasty with the other fix. Can you confirm whether or not this works correctly? #18494

[jettoblack]

Sorry, I was too hasty with the other fix. Can you confirm whether or not this works correctly? #18494

Unfortunately this did not fix either of the above cases for me.

Here's a new verbose log for the infinite loop, just in case something changed enough to provide a clue.

15182a1.txt.gz

Thanks for looking into it.