Bug: Step 3.5 Flash higher VRAM usage and loading issues after #1307

[Quairon-Nailo]

What happened?

After #1307 , loading Step 3.5 Flash with sm graph takes more VRAM than it does before #1309 , and the distribution is different, leading to issues loading unless I reduce the number of GPU layers.
https://streamable.com/6tsws9
https://streamable.com/1kbey0
My build script:

#!/bin/bash
set -e  # Exit immediately if a command exits with a non-zero status

cd /home/quair/Downloads/ik_llama.cpp/
rm -rf build
cmake -B build \
          -DGGML_CUDA=ON \
          -DCMAKE_CUDA_ARCHITECTURES=native

cmake --build build --config Release -j$(nproc)

My load script:

#!/bin/bash
llama-server \
        --host 0.0.0.0 \
        --port 8085 \
        --model "/mnt/Speed/AI/Models/AesSedai/Step-3.5-Flash-GGUF/Step-3.5-Flash-IQ4_XS-00001-of-00003.gguf" \
        -a Step3.5 \
        -b 8192 \
        -ub 8192 \
        --threads 16 \
        --ctx-size 36864 \
        --n-gpu-layers 999 \
        -ot "(1[4-9]|[0-9][0-9])\..*_exps.*=CPU" \
        --no-mmap \
        -fa on \
        -sm graph \
        -ts 180,191 \
        -np 1 \
        -smgs \
        -cram 0 \
        -cuda fusion=1

Name and Version

version: 4225 (7065488)
built with cc (GCC) 15.2.1 20260209 for x86_64-pc-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

[ikawrakow]

Instead of linking these videos, you cannot just paste the VRAM usage here? (model buffers, KV cache buffers, compute buffers)

[Nexesenex]

It might be due (at least on Qwen 3.5) to this change : fd2a709#diff-57725fa4c99fac28d0283eaa8b59c5e030355af0c8fbaf33145a64e3f2aa3420 (in the llama.cpp file).

I reverted the "llama.cpp" file related portion of the PR to avoid the creation of new buffers. I didn't mention the problem because I don't know if it's a bug or a feature, and I just edited that for myself.

[ikawrakow]

The change @Nexesenex is referring to is intentional. It shouldn't make a difference for a single-GPU system. For multi-GPU

• It allows us to know what is the GPU on which the GPU-offloaded tensors are stored for any given layer. In that way we know better what to do when a portion of the compute graph did not participate in graph split
• It puts the output tensor on the last GPU. In that way when the last repeating layer is computed one does not need to copy the result to the "main GPU", thus saving one data copy and one synchronization point.

The change should not result in any difference in model buffer sizes. It may result in different compute buffer sizes, but I found these differences not particularly significant. But then again, I don't run with batch/u-batch size of 8192.

In any case, seeing the resulting buffer sizes is much more useful than watching videos.

[Nexesenex]

@ikawrakow :

I retested with and without my revert of the llama.cpp of PR 1307. I don't have the problem of added buffers anymore, and I restored PR 1307 fully. @Quairon-Nailo , forget what I suggested. ^^

I am using the last main, minus #1326, minus PR 1310, minus PR 1303, minus 1295, plus my fork edits.

I should have been more diligent noting my steps to address the problem when it occurred.

Anyway, here's the cli command I'm using without excessive buffers problem now :

llama-server -m Qwen3.5-397B-A17B\Qwen3.5-397B-A17B-IQ4_XS-00001-of-01099.gguf -t 18 -ngl 150 -sm layer -b 512 -fdn 512 -mg 1 --device CUDA0,CUDA1,CUDA2 --max-gpu 3 -ts 23,23,15 -fa 1 -cuda fusion=1,offload-batch-size=128,mmq-id-size=128,enable-p2p=0,fa-offset=0.6931f -ot "^output.weight$=CUDA2" -ot "^blk.([6-9]|1[0-9]|2[0-2]|29|3[0-9]|4[0-5]|5[0-9]).ffn_(up|down|gate)_exps.weight$=CPU" -ot "^blk.(6|29|49).ffn_down_exps.weight$=CPU" -no-ooae -mqkv -muge -gr -ger -grt f16 -rcache --override-kv qwen35moe.expert_used_count=int:10 -ser 9,0.2 -c 196608 -ctk q6_0 -ctv q5_0 -khad --context-shift 1 --host 127.0.0.1 --port 8080 -cram 0 -cram-n-min 999999 --reasoning-tokens none

[Quairon-Nailo]

Instead of linking these videos, you cannot just paste the VRAM usage here? (model buffers, KV cache buffers, compute buffers)

You're absolutely right, sorry.
For #1309:

Estimated model buffer size per device:
    Device 0:  13837.50 MiB
    Device 1:  14241.98 MiB
llm_load_tensors: offloading 45 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 46/46 layers to GPU
llm_load_tensors:        CPU buffer size = 62077.50 MiB
llm_load_tensors:  CUDA_Host buffer size =   534.97 MiB
llm_load_tensors: CUDA_Split buffer size = 28080.01 MiB
llm_load_tensors:      CUDA0 buffer size =   534.98 MiB
................................................................................................... =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
~ggml_backend_cuda_context: have 0 graphs
.
===================================== llama_init_from_model: f16
llama_init_from_model: n_ctx         = 36864
llama_init_from_model: n_batch       = 8192
llama_init_from_model: n_ubatch      = 8192
llama_init_from_model: flash_attn    = 1
llama_init_from_model: attn_max_b    = 0
llama_init_from_model: fused_moe     = 1
llama_init_from_model: grouped er    = 0
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad    = 1
llama_init_from_model: rope_cache    = 0
llama_init_from_model: graph_reuse   = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 1
llama_init_from_model: reduce_type   = f16
llama_init_from_model: sched_async   = 0
llama_init_from_model: ser           = -1, 0
llama_init_from_model: freq_base     = 5000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: cuda_params   = fusion=1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
llama_kv_cache_init: CUDA_Split KV buffer size =  6480.07 MiB
llama_kv_cache_init: KV cache size per device:
    Device 0:  3240 MiB
    Device 1:  3240 MiB
llama_init_from_model: KV self size  = 6480.00 MiB, K (f16): 3240.00 MiB, V (f16): 3240.00 MiB
llama_init_from_model:  CUDA_Host  output buffer size =     0.49 MiB
llama_init_from_model:      CUDA0 compute buffer size =  4220.00 MiB
llama_init_from_model:      CUDA1 compute buffer size =  2624.03 MiB
llama_init_from_model:  CUDA_Host compute buffer size =  1280.09 MiB

For #1307:

Estimated model buffer size per device:
    Device 0:  13837.50 MiB
    Device 1:  14241.98 MiB
No tensors in buffer type CUDA0
llm_load_tensors: offloading 45 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 46/46 layers to GPU
llm_load_tensors:        CPU buffer size = 62077.50 MiB
llm_load_tensors:  CUDA_Host buffer size =   534.97 MiB
llm_load_tensors: CUDA_Split buffer size = 28080.01 MiB
llm_load_tensors:      CUDA1 buffer size =   534.98 MiB
................................................................................................... =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
~ggml_backend_cuda_context: have 0 graphs
.
===================================== llama_init_from_model: f16
llama_init_from_model: n_ctx         = 36864
llama_init_from_model: n_batch       = 8192
llama_init_from_model: n_ubatch      = 8192
llama_init_from_model: flash_attn    = 1
llama_init_from_model: attn_max_b    = 0
llama_init_from_model: fused_moe     = 1
llama_init_from_model: grouped er    = 0
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad    = 1
llama_init_from_model: rope_cache    = 0
llama_init_from_model: graph_reuse   = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 1
llama_init_from_model: reduce_type   = f16
llama_init_from_model: sched_async   = 0
llama_init_from_model: ser           = -1, 0
llama_init_from_model: freq_base     = 5000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: cuda_params   = fusion=1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
llama_kv_cache_init: CUDA_Split KV buffer size =  6480.07 MiB
llama_kv_cache_init: KV cache size per device:
    Device 0:  3240 MiB
    Device 1:  3240 MiB
llama_init_from_model: KV self size  = 6480.00 MiB, K (f16): 3240.00 MiB, V (f16): 3240.00 MiB
llama_init_from_model:  CUDA_Host  output buffer size =     0.49 MiB
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4156.00 MiB on device 1: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA1 buffer of size 4357881856

[ikawrakow]

And what was it before #1307 that is supposed to have increased VRAM usage? The only difference between the two logs you posted that I can see is that on #1307 it allocated 4220 MiB on CUDA0 and 2624 MiB on CUDA1 as compute buffers, while on #1309 it tried allocating 4156 MiB on CUDA1 instead, and failed (I guess, CUDA1 is the GPU driving your display).

[Quairon-Nailo]

CUDA1 is indeed the GPU driving my display. The reserved buffers in the logs seem to be the same, but the actual VRAM usage is always higher after #1307

[ikawrakow]

Then what about CUDA_VISIBLE_DEVICES=1,0 ./bin/llama-server etc.?

[Quairon-Nailo]

That allowed it to get a bit further, but it crashed immediately. Total VRAM usage for the kv cache is still about 1.3 GB above what it was before with both GPUs seeing an increase. For #1309 with CUDA_VISIBLE_DEVICES=0,1, while the model is loading, GPU0 uses 14.9 GB VRAM and GPU1 uses 16.6 GB, and in #1307 it's 15.3/16.2 GB respectively (same total use, different distribution). Meanwhile, once it loads the cache, in #1309 it uses 22.4/22.5 GB respectively, for a total of 44.9 GB, and in #1307 it uses 22.7/23.5 GB, for a total of 46.2 GB. Those numbers do include VRAM use from display and other applications, but in all cases GPU1 was using 1.8 GB before and after llama-server execution, so that should be constant.

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24103 MiB
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24126 MiB
=============================== NCCL main communicator initialized
INFO [                    main] build info | tid="140267942981632" timestamp=1772121257 build=4225 commit="70654881"
INFO [                    main] system info | tid="140267942981632" timestamp=1772121257 n_threads=16 n_threads_batch=-1 total_threads=32 system_info="AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | "
CUDA0: using device CUDA0 - 21876 MiB free
CUDA1: using device CUDA1 - 23743 MiB free
llama_model_loader: additional 2 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 54 key-value pairs and 754 tensors from /mnt/Speed/AI/Models/AesSedai/Step-3.5-Flash-GGUF/Step-3.5-Flash-IQ4_XS-00001-of-00003.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = step35
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Step 3.5 Flash
llama_model_loader: - kv   3:                         general.size_label str              = 288x7.4B
llama_model_loader: - kv   4:                            general.license str              = apache-2.0
llama_model_loader: - kv   5:                   general.base_model.count u32              = 1
llama_model_loader: - kv   6:                  general.base_model.0.name str              = Step 3.5 Flash
llama_model_loader: - kv   7:          general.base_model.0.organization str              = Stepfun Ai
llama_model_loader: - kv   8:              general.base_model.0.repo_url str              = https://huggingface.co/stepfun-ai/ste...
llama_model_loader: - kv   9:                         step35.block_count u32              = 45
llama_model_loader: - kv  10:                      step35.context_length u32              = 262144
llama_model_loader: - kv  11:                    step35.embedding_length u32              = 4096
llama_model_loader: - kv  12:                 step35.feed_forward_length u32              = 11264
llama_model_loader: - kv  13:                step35.attention.head_count arr[i32,45]      = [64, 96, 96, 96, 64, 96, 96, 96, 64, ...
llama_model_loader: - kv  14:                      step35.rope.freq_base f32              = 5000000.000000
llama_model_loader: - kv  15:                  step35.rope.freq_base_swa f32              = 10000.000000
llama_model_loader: - kv  16:                  step35.expert_gating_func u32              = 2
llama_model_loader: - kv  17:                step35.attention.key_length u32              = 128
llama_model_loader: - kv  18:              step35.attention.value_length u32              = 128
llama_model_loader: - kv  19:             step35.attention.head_count_kv arr[i32,45]      = [8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, ...
llama_model_loader: - kv  20:            step35.attention.sliding_window u32              = 512
llama_model_loader: - kv  21:    step35.attention.sliding_window_pattern arr[bool,45]     = [false, true, true, true, false, true...
llama_model_loader: - kv  22:                        step35.expert_count u32              = 288
llama_model_loader: - kv  23:                   step35.expert_used_count u32              = 8
llama_model_loader: - kv  24:          step35.expert_feed_forward_length u32              = 1280
llama_model_loader: - kv  25:   step35.expert_shared_feed_forward_length u32              = 1280
llama_model_loader: - kv  26:                step35.expert_weights_scale f32              = 3.000000
llama_model_loader: - kv  27:                 step35.expert_weights_norm bool             = true
llama_model_loader: - kv  28:           step35.leading_dense_block_count u32              = 3
llama_model_loader: - kv  29:                  step35.moe_every_n_layers u32              = 1
llama_model_loader: - kv  30:    step35.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  31:                    step35.swiglu_clamp_exp arr[f32,45]      = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  32:                  step35.swiglu_clamp_shexp arr[f32,45]      = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  33:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  34:                         tokenizer.ggml.pre str              = deepseek-v3
llama_model_loader: - kv  35:                      tokenizer.ggml.tokens arr[str,128896]  = ["<｜begin▁of▁sentence｜>", "<�...
llama_model_loader: - kv  36:                  tokenizer.ggml.token_type arr[i32,128896]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  37:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
llama_model_loader: - kv  38:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  39:                tokenizer.ggml.eos_token_id u32              = 128007
llama_model_loader: - kv  40:            tokenizer.ggml.padding_token_id u32              = 1
llama_model_loader: - kv  41:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  42:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  43:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  44:                    tokenizer.chat_template str              = {% macro render_content(content) %}{%...
llama_model_loader: - kv  45:               general.quantization_version u32              = 2
llama_model_loader: - kv  46:                          general.file_type u32              = 7
llama_model_loader: - kv  47:                      quantize.imatrix.file str              = /mnt/srv/snowdrift/fp16/Step-3.5-Flas...
llama_model_loader: - kv  48:                   quantize.imatrix.dataset str              = /mnt/srv/host/resources/KLD/calibrati...
llama_model_loader: - kv  49:             quantize.imatrix.entries_count u32              = 528
llama_model_loader: - kv  50:              quantize.imatrix.chunks_count u32              = 50
llama_model_loader: - kv  51:                                   split.no u16              = 0
llama_model_loader: - kv  52:                        split.tensors.count i32              = 754
llama_model_loader: - kv  53:                                split.count u16              = 3
llama_model_loader: - type  f32:  266 tensors
llama_model_loader: - type q8_0:  362 tensors
llama_model_loader: - type iq3_s:   84 tensors
llama_model_loader: - type iq4_xs:   42 tensors
load: printing all EOG tokens:
load:   - 128007 ('<|im_end|>')
load: special tokens cache size = 818
load: token to piece cache size = 0.8220 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = step35
llm_load_print_meta: n_ctx_train      = 262144
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 45
llm_load_print_meta: n_head           = [64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64, 96, 96, 96, 64]
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 512
llm_load_print_meta: n_swa_pattern    = 1
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = [8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8, 12, 12, 12, 8]
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11264
llm_load_print_meta: n_expert         = 288
llm_load_print_meta: n_expert_used    = 8
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 5000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 262144
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 196.956 B
llm_load_print_meta: model size       = 88.903 GiB (3.877 BPW) 
llm_load_print_meta: repeating layers = 87.858 GiB (3.852 BPW, 195.900 B parameters)
llm_load_print_meta: general.name     = Step 3.5 Flash
print_info: vocab type       = BPE
print_info: n_vocab          = 128896
print_info: n_merges         = 127741
print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
print_info: EOS token        = 128007 '<|im_end|>'
print_info: EOT token        = 128007 '<|im_end|>'
print_info: PAD token        = 1 '<｜end▁of▁sentence｜>'
print_info: LF token         = 201 'Ċ'
print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
print_info: EOG token        = 128007 '<|im_end|>'
print_info: max token length = 256
llm_load_tensors: ggml ctx size =    4.51 MiB
Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.40.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.40.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.40.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.41.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.41.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.41.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.42.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.42.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.42.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.43.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.43.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.43.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.44.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.44.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.44.ffn_down_exps.weight buffer type overriden to CPU
================================ max_gpu = 0
Estimated model buffer size per device:
    Device 0:  13837.50 MiB
    Device 1:  14241.98 MiB
No tensors in buffer type CUDA0
llm_load_tensors: offloading 45 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 46/46 layers to GPU
llm_load_tensors:        CPU buffer size = 62077.50 MiB
llm_load_tensors:  CUDA_Host buffer size =   534.97 MiB
llm_load_tensors: CUDA_Split buffer size = 28080.01 MiB
llm_load_tensors:      CUDA1 buffer size =   534.98 MiB
................................................................................................... =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
~ggml_backend_cuda_context: have 0 graphs
.
===================================== llama_init_from_model: f16
llama_init_from_model: n_ctx         = 36864
llama_init_from_model: n_batch       = 8192
llama_init_from_model: n_ubatch      = 8192
llama_init_from_model: flash_attn    = 1
llama_init_from_model: attn_max_b    = 0
llama_init_from_model: fused_moe     = 1
llama_init_from_model: grouped er    = 0
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad    = 1
llama_init_from_model: rope_cache    = 0
llama_init_from_model: graph_reuse   = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 1
llama_init_from_model: reduce_type   = f16
llama_init_from_model: sched_async   = 0
llama_init_from_model: ser           = -1, 0
llama_init_from_model: freq_base     = 5000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: cuda_params   = fusion=1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
llama_kv_cache_init: CUDA_Split KV buffer size =  6480.07 MiB
llama_kv_cache_init: KV cache size per device:
    Device 0:  3240 MiB
    Device 1:  3240 MiB
llama_init_from_model: KV self size  = 6480.00 MiB, K (f16): 3240.00 MiB, V (f16): 3240.00 MiB
llama_init_from_model:  CUDA_Host  output buffer size =     0.49 MiB
llama_init_from_model:      CUDA0 compute buffer size =  3989.53 MiB
llama_init_from_model:      CUDA1 compute buffer size =  4156.00 MiB
llama_init_from_model:  CUDA_Host compute buffer size =  1280.09 MiB
llama_init_from_model: graph nodes  = 3781
llama_init_from_model: graph splits = 333
llama_init_from_model: enabling only_active_experts scheduling
XXXXXXXX Split Mode Graph Scheduling is FORCED despite tensor overrides due to user choice.
XXXXXXXX It may or might NOT infer properly due to unsupported combinations between SMGS and every possible tensor overrides.
CUDA error: out of memory
  current device: 0, in function evaluate_and_capture_cuda_graph at /home/quair/Downloads/ik_llama.cpp/ggml/src/ggml-cuda.cu:4165
  cudaGraphInstantiate(&graph->instance, graph->graph, __null, __null, 0)
/home/quair/Downloads/ik_llama.cpp/ggml/src/ggml-cuda.cu:131: CUDA error
❌ ptrace: Operation not permitted.
❌ No stack.
❌ The program is not being run.
/home/quair/Downloads/ik_llama.cpp/Step-3.5-flash-Q5_K_M.sh: line 20: 92379 Aborted                    (core dumped) CUDA_VISIBLE_DEVICES=1,0 llama-server --host 0.0.0.0 --port 8085 --model "/mnt/Speed/AI/Models/AesSedai/Step-3.5-Flash-GGUF/Step-3.5-Flash-IQ4_XS-00001-of-00003.gguf" -a Step3.5 -b 8192 -ub 8192 --threads 16 --ctx-size 36864 --n-gpu-layers 999 -ot "(1[4-9]|[2-9][0-9])\..*_exps.*=CPU" --no-mmap -fa on -sm graph -ts 187,191 -np 1 -smgs -cram 0 -cuda fusion=1

[Quairon-Nailo]

Hold on, I see the difference, it's the compute buffer size: 4.22/2.62 GB, vs 3.99/4.16 GB (6.84 GB vs 8.15 GB), which exactly accounts for the difference I noticed.

[ikawrakow]

Of course it is the compute buffer size. The KV cache cannot possibly change.

You can try

CUDA_VISIBLE_DEVICES=1,0 ./bin/llama-server --main-gpu 1 $other_args

[Quairon-Nailo]

With that I get:

llama_init_from_model:      CUDA0 compute buffer size =  3870.53 MiB
llama_init_from_model:      CUDA1 compute buffer size =  4220.00 MiB

[ikawrakow]

Then run it with one extra MoE layer left in RAM. I would be surprised if that runs slower than what you had prior to #1307. Measure and let us know.

[Quairon-Nailo]

On #1307 , using CUDA_VISIBLE_DEVICES=1,0 and --main-gpu 1 -ot "(1[3-9]|[2-9][0-9])\..*_exps.*=CPU"

prompt eval time =   35591.91 ms / 32445 tokens (    1.10 ms per token,   911.58 tokens per second)
       eval time =   82087.00 ms /  1445 tokens (   56.81 ms per token,    17.60 tokens per second)

On #1309 using -ot "(1[4-9]|[2-9][0-9])\..*_exps.*=CPU"

prompt eval time =   33762.71 ms / 32445 tokens (    1.04 ms per token,   960.97 tokens per second)
       eval time =   62695.06 ms /  1172 tokens (   53.49 ms per token,    18.69 tokens per second)

Using the exact same prompt and locking the GPUs and memories to the same clock speeds, to ensure that conditions are equal, save for the changes mentioned above. The speed difference is not huge, but it is noticeable.