Feature Request: --n-cpu-moe option for multi GPU?

[ajunca]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add the option so --n-cpu-moe offloads the first layers from each GPU, or add the possibility to set multiple --n-cpu-moe, each starting on a different layer. Eg:
--n-cpu-moe 0:5 (start at 0 and do 5 layers)
--n-cpu-moe 10:5 (start at 10 and do 5 layters)

Motivation

If I'm not missing something, --n-cpu-moe option only offloads the first layers, but this is not very useful on multi gpu configurations where one has to offload from each GPU (eg. 5 moe layers from each gpu).

Would be nice to be able to configure this without -ot.

Thanks for the awesome work!

Possible Implementation

No response

[abc-nix]

You still need to work a bit, but use the --tensor_split to manually select the amount of layers that go to each GPU.

I have a multi-gpu system and work with --n-cpu-moe and --tensor_split to get a good use of both GPUs.

Example: I have 2 GPUs. If the model I want to use has 48 layers (47+KV cache), and I know that I can only fully offload 17 full layers to GPU2, I would use:

-ngl 99 --tensor_split 31,17 --n-cpu-moe 31

This would be the first step. I would then be removing layers from --n-cpu-moe, which would offload more experts to GPU1 until I get it filled.

The tools are there, but you need to get more involved to get the most out of them. It is way easier than the -ot option, but you still need to work a bit to optimize.

[ajunca]

Not sure to understand. How for example would you accomplish this?

--tensor-split 22,26
-ot "\.(0|1|2|3|4|5|6|7)\.ffn_(down|up|gate)_exps.=CPU"
-ot "\.(22|23|24|25|26|27|28|29)\.ffn_(down|up|gate)_exps.=CPU"

The total number of layers are 48. As I understand it, it will offload 22 layers to GPU0 and 26 layers to GPU1. Then each GPU will have its 8 first moe layer matrices offloaded to CPU.

[abc-nix]

If only 8 full layers fit in GPU1, and you can only offload 16 full layers to the GPUs, that leaves aprox 48-16=32 experts on CPU.

--tensor-split 40,8 \
--n-cpu-moe 32

But I suspect we are calculating wrong. I think paying too much attention on spreading the layers as equally as possible between both GPUs is not optimal. I have found best results if most layers (even if without experts) are in the first GPU (GPU0), and fill GPU1 with full layers (and not layers without experts). This has given me the best PP numbers.

I don't know what model, quant or GPUs you have, so you will have to experiment. Probably you can fit more than 8 full layers on GPU1, and less than 8 on GPU0 once you have found the optimal distribution.

EDIT: I'll give you an example. Let's say I have GLM-4.5 Air, that would also make 48 layers in llama.cpp. I have found that, with 64K context and KV cache quantization of Q8 I can fit 17 full layers on GPU1. The rest (48-17=31 layers) go to GPU0 without experts. If I don't offload any other expert layer to GPU0, I would have:

--tensor-split 31,17 \
--n-cpu-moe 31

Now, I start decreasing experts from the CPU (experts move automatically to GPU0), and find I can only offload 13 full experts MOE tensors to GPU0 before getting OOM errors. Then, the expert tensor left on CPU would be 31-13=18. The new distribution would be:

--tensor-split 31,17 \
--n-cpu-moe 18

[ajunca]

Oh! Thank you very much for the explanation! I think my biggest misconception was that the optimal thing to do was to split with equal layers the GPU's! Now makes more sense the --n-cpu-moe.

I will try this and see if I get an speed up.

EDIT: Sadly no speed up on my specific setup 🤷‍♂️ (probably because is a mix of GPUs + CPU). Anyway, thanks for the explanation on why --n-cpu-moe is implemented this way.

[abc-nix]

I saw you mentioned in another thread using

--batch-size 512
--ubatch-size 512

I got better PP (twice) for GLM-4.5 Air using

--batch-size 1024
--ubatch-size 1024

I had to reduce by 1 the amount of layers on one of the GPUs, but for very long contexts it is worth it. It reduces the TG speed a bit, but for me, it is worth it to reduce the PP time when modifying previous messages and "regenerating" a newer message (this is my use case, so will not apply for most users).

[ajunca]

@abc-nix I tried many combinations, my main problem now is this bug though #15112. This is why I'm trying different batch/ubatch. I will leave it but as your recommend for now.

Are you using CUDA backend? Have you encountered this issue when using >10-15k contexts?

[abc-nix]

I had to upgrade CUDA toolkit to 12.8 to get GPT-OSS to work before it was fully merged (some operations were only available in CUDA >=12.8). Now I think it was fixed, but I kept this cuda version.

I also saw that some of my GPT-OSS generations got stuck in a strange loop, but I thought it was because I was using the pre-merged branch. Even after upgrading, even with #15157 merged, I also sometimes got strange generations. I just regenerate and it usually works.

gpt-oss has issues with the template, but I think that will be fixed with #15181 plus using the official jinja template. I am running that branch right now and I don't usually get those strange generations.

GLM also has some template issues (let's hope they get merge soon). I just got a random bad generation from it while testing the IQ4 unsloth quant, but could not repeat it. I will keep an eye on the issue you linked.

[ajunca]

I also get kind of strange froze/stuck some times... I think also happen with Qwen3 235B.

I'm a bit frustrated with the "GGGGG..." issue.

[SlavikCA]

Using --n-cpu-moe is real pain on multy-GPU system.

I have RTX-3090 (24GB VRAM) + RTX-4090D (48GB VRAM).

#   Device 0: NVIDIA GeForce RTX 4090 D, compute capability 8.9, VMM: yes
#   Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes

Using GLM-4.5-Air-GGUF:UD-Q8_K_XL model

Basic params:

      --hf-repo unsloth/GLM-4.5-Air-GGUF:UD-Q8_K_XL
      --ctx-size 32768
      --flash-attn
      --jinja
      --n-gpu-layers 999

I was using this:

      -ot "blk\.([0-9]|1[0-1]|2[0-2])\.ffn=CUDA0"
      -ot "blk\.(3[0-5]|4[0-1])\.ffn=CUDA1"
      -ot exps=CPU
      # using 23.3/24 and 47.3/48 GB of VRAM

I know how to precisely add / remove layer(s) to each device in the example above.

So, I tried to switch to --n-cpu-moe:

      --tensor-split 100,24
      --n-cpu-moe 24
      # using 23.4/24  and 46/48 GB of VRAM

It works, but I had to struggle and guess, how to arrive to that params. How changing that 100,24 would affect the distribution of the layers? If I change n-cpu-moe - would that change CUDA0 or CUDA1? Hard to predict...

Wild idea: how about if I propose this:

      --n-gpu-moe 8,3

which would basically means: put 8 MOE layers to CUDA0, put 3 MOE layers to CUDA1, ... In that cause I'll not need tensor-split, n-gpu-layers, n-cpu-moe.

Also, currently there is no way to decide which CUDA device will be used for context. Right? Is it always goes to first CUDA device?

[abc-nix]

I use --tensor-split with layers numbers. If GLM-4.5 Air has 48 layers, and I place 17 full layers on GPU1, then 48-17=31 layers go to GPU0. You have to calculate backwards, that is all.
--tensor-split 31,17. It is a different amount for each model, the same as using -ot, but a bit simpler.

How changing that 100,24 would affect the distribution of the layers?

It is a proportion. If you place 100 "units" to GPU0, and 24 "units" to GPU1, the total "units" would be 124. I cannot prove this, but I suspect that --tensor-split in reality is layer-split, as I suspect it moves full layers, and not individual tensors. So, in your example, if GLM-45 Air has 48 layers, (100/124)*48~=39 (always rounding UP) are going to GPU0. If my theory is correct, --tensor-split 39,9 should also work exactly the same as your --tensor-split 100,24 command.

If I change n-cpu-moe - would that change CUDA0 or CUDA1? Hard to predict...

n-cpu-moe offloads expert tensor to CPU in order. If you use --n-cpu-moe 48 (in this example), all the 48 groups of expert tensors (from layer 0 to 47) will be offloaded to CPU. If you reduce this number, it will return to the GPU the expert groups backwards (from layer 47 to layer 0). In your example, --n-cpu-moe 24, means that the first 24 layers are offloaded to CPU and the rest go to the GPUs. That means the experts from:

• layers 39-47 (9 layers) go to GPU1
• layers 24-38 (15 layers) go to GPU0
• layers 0-23 (24 layers, the same as --n-cpu-moe 24) go to CPU.

Once you figure out the amount of full layers you can offload to GPU1, you already have the tensor-split numbers figured out. Then, you just reduce the n-cpu-moe number until GPU0 is full.

Also, currently there is no way to decide which CUDA device will be used for context. Right? Is it always goes to first CUDA device?

That is also what I understand. Unless --main-gpu does something, Device0 is where context normally goes. I always try to have my best GPU, with the best bandwidth, as Device0. You can also change the device order with the -dev option, giving it the list of device names/IDs (you can get them with the --list-devices option)..