Memory allocation on multi GPU setup

[Suppe2000]

Describe the Issue
To my setup I added a second Nvidia 4090 with 48 GB. If I run on a single GPU, I can offload a 24B Q8 model (e.g. Cydonia) with CTX of 131k, allocating around 47 GB of VRAM.

If I switch in KoboldCpp to a multi GPU setup, starting up with exact same setting, the allocation of 47 GB is shared equally between both 4090s. Theoretically I should have 96 GB of VRAM available now.
But if I slightly increase size of CTX or choose a bigger model (>24B), I run into an Out Of Memory bug.

On CUDA:
cudaMalloc failed: OOM
failed to allocate CUDA buffer

On Vulkan:
Requested buffer size exceeds device max_buffer_size limit

I sincerely do not understand what is causing it. How I can allocate the whole 96 GB of VRAM available? Any help would be appreciated.

Additional Information:
Windows 11
2 x Nvidia 4090 48GB
KoboldCpp version: any
Both CUDA / Vulkan, same result
Tensor split 1:1
"No mmap" tried without success

[askmyteapot]

Are you using "ALL" for the GPU selection?

If you launch koboldcpp from a command line, the output will stay visible so you can copy it. That will help to determine what's going wrong.

[Suppe2000]

Yes, I used "ALL" for GPU selection.
Like you asked, I started by cmd now. For comparison I checked the VRAM allocation, again.

With CTX of 131k:
CUDA0: 26,1 GB
CUDA1: 24,1 GB

With CTX of 140k:
CUDA0: 27,0 GB
CUDA1: 24,8 GB

Once I increase to 150k of CTX, Kcpp crashes, as you can see at the log, despite having lots of free VRAM available. What is causing it?

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5001 --usecuda -c 150000 -b 4096 -ngl 99 -ts 1 1
***
Welcome to KoboldCpp - Version 1.102.3
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI47802\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 53791 MB
Initializing dynamic library: koboldcpp_cublas.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=150000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, port=5001, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcontext=False, ssl=None, tensor_split=[1.0, 1.0], testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=[], usemlock=False, usemmap=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
CUDA MMQ: False
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---

Applying Tensor Split...
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:03:00.0) - 47468 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) (0000:63:00.0) - 47468 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 'ÄS'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 0 of 363
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:        CUDA0 model buffer size = 11826.45 MiB
load_tensors:        CUDA1 model buffer size = 11380.14 MiB
load_tensors:    CUDA_Host model buffer size =   680.00 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
................................................load_all_data: using async uploads for device CUDA1, buffer type CUDA1, backend CUDA1
................................................
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 150272
llama_context: n_ctx_seq     = 150272
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 4096
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (150272) > n_ctx_train (131072) -- possible training context overflow
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size = 12327.00 MiB
llama_kv_cache:      CUDA1 KV buffer size = 11153.00 MiB
llama_kv_cache: size = 23480.00 MiB (150272 cells,  40 layers,  1/1 seqs), K (f16): 11740.00 MiB, V (f16): 11740.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 2904
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 4096, n_seqs = 1, n_outputs = 1
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12710.31 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 13327728640
graph_reserve: failed to allocate compute buffers
llama_context: compute buffer allocation failed, retrying without pipeline parallelism
ggml_cuda_host_malloc: failed to allocate 2428.11 MiB of pinned memory: resource already mapped
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 2546057216
ggml_gallocr_reserve_n: failed to allocate CUDA_Host buffer of size 2546057216
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
gpttype_load_model: error: failed to load model 'E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf'
Load Text Model OK: False

Could not load text model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

And if I repeat the same on Vulkan:

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5001 --usevulkan -c 150000 -b 4096 -ngl 99 -ts 1 1
***
Welcome to KoboldCpp - Version 1.102.3
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI239922\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 53207 MB
Initializing dynamic library: koboldcpp_vulkan.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=150000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, port=5001, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcontext=False, ssl=None, tensor_split=[1.0, 1.0], testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=None, usemlock=False, usemmap=False, useswa=False, usevulkan=[], version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

Applying Tensor Split...
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) 8060S Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = NVIDIA GeForce RTX 4090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 2 = NVIDIA GeForce RTX 4090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
llama_model_load_from_file_impl: using device Vulkan1 (NVIDIA GeForce RTX 4090) (0000:63:00.0) - 80575 MiB free
llama_model_load_from_file_impl: using device Vulkan2 (NVIDIA GeForce RTX 4090) (0000:03:00.0) - 80575 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 '─è'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
Apply NVIDIA Anti-BSOD Fix for KCPP
Apply NVIDIA Anti-BSOD Fix for KCPP
load_tensors: relocated tensors: 0 of 363
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:      Vulkan1 model buffer size = 11826.45 MiB
load_tensors:      Vulkan2 model buffer size = 11380.14 MiB
load_tensors:  Vulkan_Host model buffer size =   680.00 MiB
................................................................................................
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 150272
llama_context: n_ctx_seq     = 150272
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 4096
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (150272) > n_ctx_train (131072) -- possible training context overflow
set_abort_callback: call
llama_context: Vulkan_Host  output buffer size =     0.50 MiB
llama_kv_cache:    Vulkan1 KV buffer size = 12327.00 MiB
llama_kv_cache:    Vulkan2 KV buffer size = 11153.00 MiB
llama_kv_cache: size = 23480.00 MiB (150272 cells,  40 layers,  1/1 seqs), K (f16): 11740.00 MiB, V (f16): 11740.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 2904
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 4096, n_seqs = 1, n_outputs = 1

WARNING: Requested buffer size (2546057248) exceeds device max_buffer_size limit (2147483648)!
ggml_vulkan: Failed to allocate pinned memory (vk::Device::allocateMemory: ErrorOutOfDeviceMemory)
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 2546057248
ggml_gallocr_reserve_n: failed to allocate Vulkan_Host buffer of size 2546057216
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
gpttype_load_model: error: failed to load model 'E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf'
Load Text Model OK: False

Could not load text model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

Would it be wrong to assume that with a CTX size of 150k the allocation of VRAM should be around 28 GB & 26 GB on 24B Cydonia Q8 ?

[LostRuins]

So the failure is here:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12710.31 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 13327728640

When allocating the kv buffer the allocation runs into a out of memory error. I'm not sure if it's because the allocation size itself is too large, or whether you are indeed out of memory. The model buffer itself allocated 11826.45 mb fine, but that allocation is smaller than the KV buffer.

I assume you are using modified cards? Are these the stock drivers or modified drivers? Already I notice something very wonky - CUDA thinks you have 47GB of VRAM available per card while Vulkan thinks you have 80GB of VRAM per card, so something is very weird already even before starting the load.

Here are 4 approaches you can try:

• Can you try loading in row split mode instead? I'm not sure if it will make a difference but multigpu is always finnicky.
• If that fails, Can you change your tensor_split from 1.0 1.0 to 1.0 1.5, then try again and check the logs. I wanna see if we leave more room on GPU0 whether the allocation for the KV bufferwill still fail.
• If this still fails, you can consider other workarounds. Run with --quantkv to quantize the KV cache which should greatly reduce the VRAM use at massive contexts.
• Finally, you can try enabling SWA, which can help reduce kv cache memory usage too.

[Suppe2000]

I'm not sure if it's because the allocation size itself is too large, or whether you are indeed out of memory.

I am definitley not out of memory. That is the point of this bug report. But obviously I do not understand either what is going on.

I assume you are using modified cards? Are these the stock drivers or modified drivers?

Yes, these are modded cards from Shenzhen to double the VRAM. The firmware is modified. The drivers are stock.
As I said earlier, I can load without problems on a single card the 24B 8Q with 131k CTX, allocating about 47 GB of VRAM. So the cards work fine individually. The problem just occurs in a multi GPU setup.

Already I notice something very wonky - CUDA thinks you have 47GB of VRAM available per card while Vulkan thinks you have 80GB of VRAM per card, so something is very weird already even before starting the load.

I might misunderstand, but this seems to be fully correct to me. Under CUDA it reports the available VRAM for CUDA access, which is the 48GB of VRAM per card, only. But the available shown space for Vulkan is the addressed RAM space (48 GB of VRAM + 32 GB of shared RAM).

Can you try loading in row split mode instead? I'm not sure if it will make a difference but multigpu is always finnicky.

Sure, crashing again with OOM:

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5001 --usecuda rowsplit -c 150000 -b 4096 -ngl 99
***
Welcome to KoboldCpp - Version 1.102.3
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI85802\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 52502 MB
Initializing dynamic library: koboldcpp_cublas.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=150000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, port=5001, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcontext=False, ssl=None, tensor_split=None, testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=['rowsplit'], usemlock=False, usemmap=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
CUDA MMQ: True
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:03:00.0) - 47468 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) (0000:63:00.0) - 47468 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 'ÄS'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 81 of 363
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:        CUDA0 model buffer size =     0.82 MiB
load_tensors:  CUDA0_Split model buffer size = 11825.62 MiB
load_tensors:        CUDA1 model buffer size =     0.76 MiB
load_tensors:  CUDA1_Split model buffer size = 11379.38 MiB
load_tensors:    CUDA_Host model buffer size =   680.00 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
................................................load_all_data: using async uploads for device CUDA1, buffer type CUDA1, backend CUDA1
................................................
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 150272
llama_context: n_ctx_seq     = 150272
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 4096
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (150272) > n_ctx_train (131072) -- possible training context overflow
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size = 12327.00 MiB
llama_kv_cache:      CUDA1 KV buffer size = 11153.00 MiB
llama_kv_cache: size = 23480.00 MiB (150272 cells,  40 layers,  1/1 seqs), K (f16): 11740.00 MiB, V (f16): 11740.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 2904
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 4096, n_seqs = 1, n_outputs = 1
ggml_cuda_host_malloc: failed to allocate 2428.11 MiB of pinned memory: resource already mapped
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 2546057216
ggml_gallocr_reserve_n: failed to allocate CUDA_Host buffer of size 2546057216
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
gpttype_load_model: error: failed to load model 'E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf'
Load Text Model OK: False

Could not load text model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

If that fails, Can you change your tensor_split from 1.0 1.0 to 1.0 1.5, then try again and check the logs. I wanna see if we leave more room on GPU0 whether the allocation for the KV bufferwill still fail.

No problem. But it is crashing again:

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5001 --usecuda -c 150000 -b 4096 -ngl 99 -ts 1.0 1.5
***
Welcome to KoboldCpp - Version 1.102.3
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI213322\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 53136 MB
Initializing dynamic library: koboldcpp_cublas.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=150000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, port=5001, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcontext=False, ssl=None, tensor_split=[1.0, 1.5], testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=[], usemlock=False, usemmap=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
CUDA MMQ: False
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---

Applying Tensor Split...
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:03:00.0) - 47468 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) (0000:63:00.0) - 47468 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 'ÄS'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 0 of 363
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:        CUDA0 model buffer size =  9573.79 MiB
load_tensors:        CUDA1 model buffer size = 13632.79 MiB
load_tensors:    CUDA_Host model buffer size =   680.00 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
.......................................load_all_data: using async uploads for device CUDA1, buffer type CUDA1, backend CUDA1
..........................................................
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 150272
llama_context: n_ctx_seq     = 150272
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 4096
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (150272) > n_ctx_train (131072) -- possible training context overflow
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size =  9979.00 MiB
llama_kv_cache:      CUDA1 KV buffer size = 13501.00 MiB
llama_kv_cache: size = 23480.00 MiB (150272 cells,  40 layers,  1/1 seqs), K (f16): 11740.00 MiB, V (f16): 11740.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 2904
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 4096, n_seqs = 1, n_outputs = 1
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12710.31 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 13327728640
graph_reserve: failed to allocate compute buffers
llama_context: compute buffer allocation failed, retrying without pipeline parallelism
ggml_cuda_host_malloc: failed to allocate 2428.11 MiB of pinned memory: resource already mapped
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 2546057216
ggml_gallocr_reserve_n: failed to allocate CUDA_Host buffer of size 2546057216
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
gpttype_load_model: error: failed to load model 'E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf'
Load Text Model OK: False

Could not load text model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

If this still fails, you can consider other workarounds. Run with --quantkv to quantize the KV cache which should greatly reduce the VRAM use at massive contexts.
Finally, you can try enabling SWA, which can help reduce kv cache memory usage too.

I know these methods. I do not wish to reduce VRAM or cache usage. That won't fix the problem for me. I want to figure out how I can fully allocate my available 96 GB of VRAM among both my cards.

Just to figure out what is going on, on the same system I loaded the model with same settings into my iGPU (Halo Strix 128 GB with 64 RAM : 64 VRAM split in UEFI). The model loads successfully, but shows same buffer limit bug(?):

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5001 --usevulkan 0 -c 150000 -b 4096 -ngl 99 -ts 1 1
***
Welcome to KoboldCpp - Version 1.102.3
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI80162\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 53117 MB
Initializing dynamic library: koboldcpp_vulkan.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=150000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, port=5001, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcontext=False, ssl=None, tensor_split=[1.0, 1.0], testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=None, usemlock=False, usemmap=False, useswa=False, usevulkan=[0], version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

Applying Tensor Split...
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) 8060S Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon(TM) 8060S Graphics) (unknown id) - 93217 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 '─è'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 0 of 363
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:      Vulkan0 model buffer size = 23206.58 MiB
load_tensors:  Vulkan_Host model buffer size =   680.00 MiB
.................................................................................................
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 150272
llama_context: n_ctx_seq     = 150272
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 4096
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (150272) > n_ctx_train (131072) -- possible training context overflow
set_abort_callback: call
llama_context: Vulkan_Host  output buffer size =     0.50 MiB
llama_kv_cache:    Vulkan0 KV buffer size = 23480.00 MiB
llama_kv_cache: size = 23480.00 MiB (150272 cells,  40 layers,  1/1 seqs), K (f16): 11740.00 MiB, V (f16): 11740.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
llama_context: max_nodes = 2904
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 4096, n_seqs = 1, n_outputs = 1

WARNING: Requested buffer size (2462056448) exceeds device max_buffer_size limit (2147483648)!

WARNING: Requested buffer size (2546057248) exceeds device max_buffer_size limit (2147483648)!
llama_context:    Vulkan0 compute buffer size =  3778.09 MiB
llama_context: Vulkan_Host compute buffer size =  2428.11 MiB
llama_context: graph nodes  = 1247
llama_context: graph splits = 2
Threadpool set to 15 threads and 15 blasthreads...
attach_threadpool: call
Starting model warm up, please wait a moment...
Load Text Model OK: True
Chat completion heuristic: Mistral Tekken
Embedded KoboldAI Lite loaded.
Embedded API docs loaded.
Llama.cpp UI loaded.
======
Active Modules: TextGeneration
Inactive Modules: ImageGeneration VoiceRecognition MultimodalVision MultimodalAudio NetworkMultiplayer ApiKeyPassword WebSearchProxy TextToSpeech VectorEmbeddings AdminControl
Enabled APIs: KoboldCppApi OpenAiApi OllamaApi
Starting Kobold API on port 5001 at http://0.0.0.0:5001/api/
Starting OpenAI Compatible API on port 5001 at http://0.0.0.0:5001/v1/
Starting llama.cpp secondary WebUI at http://0.0.0.0:5001/lcpp/
======
Please connect to custom endpoint at http://0.0.0.0:5001

Sadly, it seems to be impossible to offload to both iGPU and dGPUs at the same time. It would be useful for MoEs with fast RAM (e.g. Halo Strix). But that topic is unrelated, of course.

However, I keep thinking, might the OOM bug related to the shared RAM memory? That somehow an address overlap happens? Inside the error message is written "resource already mapped", which seems to be the case? I am just wildly guessing.

[LostRuins]

So it's not an address overlap. What is happening is that the original allocation with pipeline parallelism fails (12GB alloc), and then it turns off pipeline parallelism and tries to reallocate (2.5GB) but the address has already been mapped and failed to unmap, causing everything to fail (unrelated issue).

The issue is likely that pipeline parallelism with 4 copies results in individual buffer allocation sizes that are beyond CUDA's limit for your case (the allocation must be one continuous block).

I am preparing a test version with pipeline parallelism reduced to 2 copies at https://github.com/LostRuins/koboldcpp/actions/runs/20124210895 , can you give that build a try with your original default settings and see if that works for you?

[Suppe2000]

can you give that build a try with your original default settings and see if that works for you?

Thank you. This creates some follow-up questions for me. But lets first check on the result:

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5001 --usecuda -c 150000 -b 4096 -ngl 99 -ts 1 1
***
Welcome to KoboldCpp - Version 1.104
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI49562\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 53652 MB
Initializing dynamic library: koboldcpp_cublas.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=150000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, port=5001, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcache=False, smartcontext=False, ssl=None, tensor_split=[1.0, 1.0], testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=[], usemlock=False, usemmap=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
CUDA MMQ: False
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---

Applying Tensor Split...
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:03:00.0) - 47468 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) (0000:63:00.0) - 47468 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 'ÄS'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 0 of 363
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:        CUDA0 model buffer size = 11826.45 MiB
load_tensors:        CUDA1 model buffer size = 11380.14 MiB
load_tensors:    CUDA_Host model buffer size =   680.00 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
................................................load_all_data: using async uploads for device CUDA1, buffer type CUDA1, backend CUDA1
................................................
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 150272
llama_context: n_ctx_seq     = 150272
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 4096
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (150272) > n_ctx_train (131072) -- possible training context overflow
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size = 12327.00 MiB
llama_kv_cache:      CUDA1 KV buffer size = 11153.00 MiB
llama_kv_cache: size = 23480.00 MiB (150272 cells,  40 layers,  1/1 seqs), K (f16): 11740.00 MiB, V (f16): 11740.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 2904
llama_context: pipeline parallelism enabled (n_copies=2)
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 4096, n_seqs = 1, n_outputs = 1
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4636.19 MiB on device 1: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA1 buffer of size 4861394944
graph_reserve: failed to allocate compute buffers
llama_context: compute buffer allocation failed, retrying without pipeline parallelism
llama_context:      CUDA0 compute buffer size =  3810.08 MiB
llama_context:      CUDA1 compute buffer size =  2934.09 MiB
llama_context:  CUDA_Host compute buffer size =  2428.11 MiB
llama_context: graph nodes  = 1247
llama_context: graph splits = 3
Threadpool set to 15 threads and 15 blasthreads...
attach_threadpool: call
Starting model warm up, please wait a moment...
CUDA error: out of memory
  current device: 1, in function alloc at D:\a\koboldcpp\koboldcpp\ggml\src\ggml-cuda\ggml-cuda.cu:470
  cuMemCreate(&handle, reserve_size, &prop, 0)
D:\a\koboldcpp\koboldcpp\ggml\src\ggml-cuda\ggml-cuda.cu:94: CUDA error

As you can see, with n=2 pipeline parallelism this time, it failed again, but he managed to load the model without pipeline parallelism. But during the warmup he crashed with OOM again. Just during the warmup I glimpsed into the VRAM allocation, which was actually:
CUDA0: 28,1 of 48 GB
CUDA1: 25,4 of 48 GB
So there was more than enough VRAM left as available unused space.

What is happening is that the original allocation with pipeline parallelism fails (12GB alloc)

There is more than 12 GB available on CUDA0. What happens? Why does it fail?

but the address has already been mapped and failed to unmap

Why does he tries to map so much memory beforehand? With CTX size of 131k, my memory usage during PP or TG is not increasing. It keeps equal.
So for what is the reservation? Can it be more conservative or possibly dynamic?

The issue is likely that pipeline parallelism with 4 copies results in individual buffer allocation sizes that are beyond CUDA's limit for your case

Won't it inflict the PP or TG speed negativley in CUDA? And what is with the same problem on Vulkan?
What if we try it with pipeline parallelism disabled in the first place? How does this will inflict speeds?

In general, I still don't get it. The problem does not even exist with a single GPU. If I run 131k CTX size with same setting, I can allocate without problems the whole space of available VRAM without crashing during warmup or PP/TG. Why it is so different with a mGPU setup?
Isn't pipeline paralleism with n=4 standard in CUDA, even with a single GPU? I am no expert of course, but eager to understand and learn.

Thanks so far for your support. I really appreciate your effort and am eager to try more of your solutions. :)

[LostRuins]

On a single GPU, there is no pipeline parallelism. It was added here ggml-org#6017

Looks like I may have to add a flag to turn it off, since the fallback in ggml-org#16748 is not working.

[LostRuins]

Alright I've moved pipeline parallelism out to a flag, and disabled it by default. Please try this build https://github.com/LostRuins/koboldcpp/actions/runs/20134050330

[Suppe2000]

Thank you. I tried it with same parameters, but it is failing again, sadly. Ressource already mapped.

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5001 --usecuda -c 150000 -b 4096 -ngl 99 -ts 1 1
***
Welcome to KoboldCpp - Version 1.104
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI17402\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 52477 MB
Initializing dynamic library: koboldcpp_cublas.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=150000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, pipelineparallel=False, port=5001, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcache=False, smartcontext=False, ssl=None, tensor_split=[1.0, 1.0], testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=[], usemlock=False, usemmap=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
CUDA MMQ: False
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---

Applying Tensor Split...
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:03:00.0) - 47468 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) (0000:63:00.0) - 47468 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 'ÄS'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 0 of 363
load_tensors: offloading 40 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:        CUDA0 model buffer size = 11826.45 MiB
load_tensors:        CUDA1 model buffer size = 11380.14 MiB
load_tensors:    CUDA_Host model buffer size =   680.00 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
................................................load_all_data: using async uploads for device CUDA1, buffer type CUDA1, backend CUDA1
................................................
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 150272
llama_context: n_ctx_seq     = 150272
llama_context: n_batch       = 4096
llama_context: n_ubatch      = 4096
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (150272) > n_ctx_train (131072) -- possible training context overflow
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size = 12327.00 MiB
llama_kv_cache:      CUDA1 KV buffer size = 11153.00 MiB
llama_kv_cache: size = 23480.00 MiB (150272 cells,  40 layers,  1/1 seqs), K (f16): 11740.00 MiB, V (f16): 11740.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 2904
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 4096, n_seqs = 1, n_outputs = 1
ggml_cuda_host_malloc: failed to allocate 2428.11 MiB of pinned memory: resource already mapped
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 2546057216
ggml_gallocr_reserve_n: failed to allocate CUDA_Host buffer of size 2546057216
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
gpttype_load_model: error: failed to load model 'E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf'
Load Text Model OK: False

Could not load text model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

It seems to me, I might be wrong, that that the pipeline parallelism isn't the problem, but a symptom, maybe. I tried something else now. I started on CUDA dev0 single GPU setup same settings, but CTX size of 100k. Once successfully loaded, 41,1 GB of VRAM is used. Than I try to start on CUDA dev1 single GPU setup with same setup. But the loading fails with an OOM error, despite having all 48 GB of VRAM free. I can repeat the process in reverse configuration (first dev1 and than dev0) having same result.

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5002 --usecuda 1 -c 100000 -b 4096 -ngl 99
***
Welcome to KoboldCpp - Version 1.104
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI197282\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 49500 MB
Initializing dynamic library: koboldcpp_cublas.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=100000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, pipelineparallel=False, port=5002, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcache=False, smartcontext=False, ssl=None, tensor_split=None, testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=['1'], usemlock=False, usemmap=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
CUDA MMQ: True
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:63:00.0) - 47468 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 'ÄS'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 0 of 363
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 23206.58 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 24333864960
llama_model_load: error loading model: unable to allocate CUDA0 buffer
llama_model_load_from_file_impl: failed to load model
Traceback (most recent call last):
  File "koboldcpp.py", line 8475, in <module>
    main(launch_args=parser.parse_args(),default_args=parser.parse_args([]))
  File "koboldcpp.py", line 7453, in main
    kcpp_main_process(args,global_memory,using_gui_launcher)
  File "koboldcpp.py", line 7911, in kcpp_main_process
    loadok = load_model(modelname)
  File "koboldcpp.py", line 1527, in load_model
    ret = handle.load_model(inputs)
OSError: exception: access violation reading 0x0000000000000004
[2140] Failed to execute script 'koboldcpp' due to unhandled exception!

I can not even load a model with Vulkan on the second device GPU, as long as the first GPU has an active model loaded. But it becomes even funnier. Now with an active model loaded on CUDA dev0, I tried to load a model on Vulkan dev0 (Halo Strix iGPU with 64 GB VRAM in UEFI), resulting again in a OOM bug. Why? Again, if I shut down the loaded model on CUDA dev0, Kccp with Vulkan can load the model without problems on the iGPU, too.

F:\AI>F:\AI\koboldcpp.exe -m E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf -fa --host 0.0.0.0 --port 5002 --usevulkan 0 -c 100000 -b 4096 -ngl 99
***
Welcome to KoboldCpp - Version 1.104
Loading Chat Completions Adapter: C:\Users\...\AppData\Local\Temp\_MEI6682\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.26200 AMD64 AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD
Detected Available GPU Memory: 49140 MB
Detected Available RAM: 49700 MB
Initializing dynamic library: koboldcpp_vulkan.dll
==========
Namespace(admin=False, admindir='', adminpassword=None, analyze='', batchsize=4096, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=100000, debugmode=0, defaultgenamt=768, draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=True, forceversion=0, foreground=False, genlimit=0, gpulayers=99, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='0.0.0.0', ignoremissing=False, jinja=False, jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mmproj='', mmprojcpu=False, model=['E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf'], model_param='E:\\AI\\LLM\\Cydonia-24B-v4zk-Q8_0.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, nommap=False, nomodel=False, noshift=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', password=None, pipelineparallel=False, port=5002, port_param=5001, preloadstory='', prompt='', quantkv=0, quiet=False, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], savedatafile='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults='', sdlora='', sdloramult=1.0, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=0, sdtiledvae=768, sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcache=False, smartcontext=False, ssl=None, tensor_split=None, testmemory=False, threads=15, ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', useclblast=None, usecpu=False, usecuda=None, usemlock=False, usemmap=False, useswa=False, usevulkan=[0], version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) 8060S Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon(TM) 8060S Graphics) (unknown id) - 93217 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 363 tensors from E:\AI\LLM\Cydonia-24B-v4zk-Q8_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 23.33 GiB (8.50 BPW)
init_tokenizer: initializing tokenizer for type 2
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_embd_inp       = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Cydonia 24B V4Zk Workspace
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 '─è'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 0 of 363
ggml_vulkan: Device memory allocation of size 1002782720 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
alloc_tensor_range: failed to allocate Vulkan0 buffer of size 1002782720
llama_model_load: error loading model: unable to allocate Vulkan0 buffer
llama_model_load_from_file_impl: failed to load model
Traceback (most recent call last):
  File "koboldcpp.py", line 8475, in <module>
    main(launch_args=parser.parse_args(),default_args=parser.parse_args([]))
  File "koboldcpp.py", line 7453, in main
    kcpp_main_process(args,global_memory,using_gui_launcher)
  File "koboldcpp.py", line 7911, in kcpp_main_process
    loadok = load_model(modelname)
  File "koboldcpp.py", line 1527, in load_model
    ret = handle.load_model(inputs)
OSError: exception: access violation reading 0x0000000000000004
[25536] Failed to execute script 'koboldcpp' due to unhandled exception!

You can see that both the CUDA dev1 and the Vulkan iGPU both had free VRAM space.

[Suppe2000]

Okay, things get even more weird now, as I tried to mess around a bit.
All the logs above are with 2 dGPUs on a HaloStrix system with 128GB of RAM. Inside of UEFI this RAM is setted on manual to a 1:1 split, meaning 64 GB of RAM are dedicated as VRAM for the iGPU. Under this settings I can successfully load, as mentioned in the beginning, a 24B Q8 model with 131 CTX size on the CUDA devices, or I can load 150 CTX size Vulkan on the iGPU.

I went to UEFI now and changed the switch of share. Now 32 GB are dedicated to RAM and 96 GB VRAM of the iGPU. As a result I can no more load a CTX size of 150k into the iGPU, despite having 50% more VRAM available (OOM error). But more funny, I can also no more load a CTX size of 131k into any of the dGPU (OOM error again). How is that even related? To me it does not make any sense.

However, I turned that around, went back to UEFI and gave the iGPU 512 MB of VRAM only, while putting the rest back to RAM space. Back in Windows I suddenly can load bigger CTX size into the iGPU ! But not that, I fired up my CUDA multi GPU setup, fiddled a bit with the settings, and managed to pack a whopping 250k CTX size, successfully loading the model.

Thank you for helping me find the cause. However, I sincerely do not get it. Why is the RAM / VRAM split of the HaloStrix system messing with my addressable space on my CUDA devices? In any case, this shouldn't make a difference. Very weird.

With 250k CTX size and ts 1.0:1.5:

pipeline parallelism n=4
CUDA0 45,7 GB used VRAM
CUDA1 46,1 GB used VRAM

pipeline parallelism n=2
CUDA0 37,8 GB used VRAM
CUDA1 42,1 GB used VRAM

pipeline parallelism OFF
CUDA0 32,4 GB used VRAM
CUDA1 39,7 GB used VRAM

Benchmarked with 120k CTX size, only single runs.

pipeline parallelism n=4
ProcessingTime: 47.898s
ProcessingSpeed: 2503.24T/s
GenerationTime: 6.841s
GenerationSpeed: 14.62T/s
TotalTime: 54.739s

pipeline parallelism n=2
ProcessingTime: 48.310s
ProcessingSpeed: 2481.89T/s
GenerationTime: 5.554s
GenerationSpeed: 18.01T/s
TotalTime: 53.864s

pipeline parallelism OFF
ProcessingTime: 52.600s
ProcessingSpeed: 2279.47T/s
GenerationTime: 5.558s
GenerationSpeed: 17.99T/s
TotalTime: 58.158s

From this values it would be cool, if inside of Kccp it is possible to change the value of pipeline parallelism or to switch it off, if desired. The speed gain with parallelism is marginal from 2 to 4 (at least for me), while even the TG slows down. However, the VRAM usage massivley drops with 2 or OFF settings, which can help to fit bigger models or CTX size. So everyone can find their personal preferred values, depending on their GPU configurations.

[LostRuins]

Ah good to know you got it to work. I agree pipeline parallelism off by default is better then. It will be an opt-in toggle next version.

[wbruna]

However, I turned that around, went back to UEFI and gave the iGPU 512 MB of VRAM only, while putting the rest back to RAM space

Possibly related: ggml-org#12770

[LostRuins]

This is now added in the latest release, toggle with --pipelineparallel