Split model between NUMA nodes for CPU inference

[CommanderLake]

If llama.cpp can split a model between GPUs can it split a model between NUMA nodes for CPU inference to eliminate inter-node memory access and take full advantage of the available memory bandwidth?
I ask this because i'm trying to optimize a llama.cpp build for a Xeon Max 9480 and regardless of what i enable in the build, OpenMP, AMX, NUMA etc. there are diminishing returns beyond a single tile worth of threads (28) when in theory if inter-node memory access could be minimised or eliminated by splitting the model evenly and isolating the threads to the correct nodes it could scale far better.

[wkgcass]

You may check this one: #12289
Instead of splitting computing and memory, it would be much easier to copy the model to multiple numa nodes.

The above modification is already implemented and opensourced.

[CommanderLake]

Not if i want to use only the HBM for optimal performance, doing that would limit me to a 16GB model which means no 32B Q4_K_M or above.

[usrlocalben]

Also follow #11733

[CommanderLake]

But how do i use this workaround in my own code using common and llama function calls?
I just tried several methods that might mimic this workaround but it is still not scaling well on more than one tile on the 9480.

[usrlocalben]

And #12088

[CommanderLake]

NUMA and SNC cannot be disabled on the Xeon Max with only HBM.

[CommanderLake]

Using the windows performance counter "NUMA Node Memory\Available MBytes" i can monitor how the memory is allocated and even with the NUMA strategy set to "Distribute" i can clearly see it filling up ONE node and only spilling over to the others when its full.

However, if i use "_putenv("OMP_PROC_BIND=close");" to spread the threads evenly across all NUMA nodes the context or kv cache gets allocated evenly also and its only the model weights that dont get distributed!

[CommanderLake]

I noticed that QwQ 32B allocates most of the memory with a single thread and performance is much worse than the Deepseek R1 Qwen 32B model which seems to allocate most of the memory with multiple threads which distributes the memory more evenly between nodes, any ideas why this might be?

[CommanderLake]

Is anyone working on this, i honestly dont understand why a model cant be split between NUMA nodes like with multiple GPUs?

[CommanderLake]

So there's no hope for getting optimal performance out of a Xeon Max for models that need to be split across all 4 nodes but cannot be copied due to the limited HBM capacity?

[slaren]

I don't think there is anything stopping you from dividing the rows in each weight matrix between the different NUMA nodes, and then scheduling the work so that only the threads belonging to the NUMA nodes that own the rows work of that part of the matrix multiplication. That should be similar to what happens when using mmap so that the first node to touch the memory region is where it is allocated, just more reliable and taking into account page sizes. Other operations may be harder to divide across NUMA nodes, but they are not very important in comparison.

[JohannesGaessler]

One of my next goals will be to get tensor parallelism for GPUs to work in a backend-agnostic way. In principle it would then also be possible to apply the same code for multiple NUMA nodes.

[NoureddineBouhali]

@slaren . Wouldn't GEMM kernels be divided into submatrices, each submatrice matmul is executed in a single core's simd unit. The cumulative bandwidth would then depend on where each matrice's weights are allocated beforehand (in "llama_model::load_tensors"). Then, it would all depend if weights were transfered to corresponding thread's home numa node, or were previously loaded due to a precedant first touch policy from llama.cpp (that am not aware of yet.)

[CommanderLake]

o3's 2 cents...

FYI: the memkind / hbwmalloc allocator that ships with Intel oneAPI already lets you put each allocation straight into the local HBM NUMA node the calling thread is pinned to (hbw_malloc(), hbw_posix_memalign_psize(), etc.).
If ggml_aligned_malloc() were conditionally swapped for those calls (with HBW_POLICY_BIND so it errors instead of spilling to DDR) and the loader threads were pinned round-robin, every tensor slice would land in the right 16 GB bank without needing custom mmap + mbind plumbing. Could be a quick prototype before a full CPU-shard backend.

[CommanderLake]

Still no NUMA love?

[SmallAndSoft]

Have you looked at what can be done with RPC workers in llama.cpp? Maybe you can pin several of them right where you need them and make them communicate via loopback or unix socket (preferably).

[ggerganov]

Worth a try.

[CommanderLake]

Hey i heard MoE models can have the experts split between GPUs, can that work the same for NUMA nodes?

[saood06]

Hey i heard MoE models can have the experts split between GPUs, can that work the same for NUMA nodes?

In theory yes. But I'm not sure if there are any known implementations.

There is also the potential roadblock that the experts in a GGUF are generally consolidated (this is not the case with most safetensors, which you can see by using Huggingfaces viewing tools).

[usrlocalben]

See recent development on this from Intel x sglang.

However, IMO their wording throughout the report and documentation is weaselly. They write about "CPU" and that it's "optimized for" but as currently implemented it requires AMX for MLA so you can't make progress without it even if you avoid FP8.

[kassieclaire]

Xeon Max 9480 has AMX (https://www.intel.com/content/www/us/en/products/sku/232592/intel-xeon-cpu-max-9480-processor-112-5m-cache-1-90-ghz/specifications.html), so from what I understand, the sglang implementation should work on that. I'll give it a spin on my dual 8642y+ and 768GB DDR5 (across two sockets) to see how it performs compared to llama.cpp/k_llama.cpp. I'm coming from trying to get ktransformers to work for larger models and failing (it's really not built for normal use, moreso benchmarks). Tensor parallelism on NUMA that the Intel implementation on SG-Lang does seems like what I'd want for very large models with Q8 quants.

[CommanderLake]

Llama.cpp has AMX and i build it with AMX for use with my 9480 but that doesnt fix the memory access issues you can see in the below post.

[kassieclaire]

Llama.cpp has AMX and i build it with AMX for use with my 9480 but that doesnt fix the memory access issues you can see in the below post.

Oh, I meant that I'm interested in if the sglang engine from Intel works, which is separate from llama.cpp. Since it uses tensor parallel on CPUs, it may not require 2x RAM for NUMA parallelism. If that implementation works very well it could be a reason to implement tensor parallel over Numa nodes on llama.cpp.

[CommanderLake]

Here's some inspiration...

Screenshot from Intel VTune is attached—this is my Xeon Max 9480 (4 tiles × 14 cores, 64 GB HBM, no DDR) running llama.cpp inside my own Windows front‑end LM Stud, though i have not released a version with these instruction sets.

• Built llama.cpp as a DLL with the Intel 2025 compiler, switch /Qxsapphirerapids (so AVX‑512 + VNNI + AMX are all in play)
• OpenMP, 56 threads (= physical cores)
• NUMA strategy set at start‑up with llama_numa_init(GGML_NUMA_STRATEGY_DISTRIBUTE)
• OMP_PROC_BIND=close is hard‑wired in my program’s BackendInit()
• Model is Qwen3‑32B‑UD‑Q4_K_XL

What VTune says ~10 k tokens in

• Cache‑Memory Bandwidth is stuck at 100 % → the EMIB links between tiles are saturated
• ~48 % Memory‑Bound, mostly L3‑latency stalls (traffic is remote‑L3, not HBM)
• Data‑Sharing ~12 %, so pages keep bouncing between tiles
• Worst part: as context grows, tokens‑per‑second nosedives—KV‑cache pages clearly end up on the wrong NUMA node on top of the weights already being lopsided, if inference starts at 13 tok/s it will be chugging along at 4 at 10 k tokens

[usrlocalben]

If you need throughput at e.g 50K, 100K ctx then you need hybrid inference.

tok/sec will always nosedive since attention has quadratic complexity. GPUs "solve" this by pushing the curve out at least towards the context limit of the model. EPYC/Xeon platforms have a lot of bandwidth for experts/FFN/MoE, but they can't match a GPU at attention. To see this, do a sweep-bench and chart the values in sec/token instead of token/sec:

This is K2 (Q8+Q4/Q6 moe) on 2S EPYC 9115 w/24x DDR5, ik_llama.

[CommanderLake]

Kinda depressing seeing the AI Ryzen MAX+ 395 with just 256GB/s of memory bandwidth generate twice as fast as the Xeon Max 9480 with 800GB/s of usable bandwidth.

[usrlocalben]

Have you tried the sglang Intel implementation ? They claim 85% memory efficiency w/tensor parallelism via NUMA.

[kassieclaire]

Have you tried the sglang Intel implementation ? They claim 85% memory efficiency w/tensor parallelism via NUMA.

I've been using this implementation, it's interesting since it does show how fast the CPU-only inference can run, but there are a few issues with it. Those being:

1. No hybrid
2. Very limited quantization support on CPU (INT8, bf16, fp8) -- with slow development towards support of other quantization (INT4 WoQ, FP4 repacking to INT8 for AMX)
3. Development for support of various model features is not across platforms, and usually CPU development is completely separate -- newer models like GLM-4.5 still don't work on CPU from my tests.

It's promising, but unfortunately isn't getting the love it needs. CPU support still feels more like a "demo" in the realm of k-transformers than a reliable inference backend.

[CommanderLake]

I need it to be in native code on Windows so i can integrate it with my WinForms project LM Stud, is anyone working on tensor parallelism for Llama.cpp? I'd love to be able to do it but there's no way i could figure it out.

[usrlocalben]

Two other projects that claim NUMA-awareness that I've seen recently are

• lktransformers, and
• fastllm

I tried lktransformers, but so far did not get inspiring results. You may have a better outcome. It's derived from ktransformers, so there is the opportunity for FP8-attention on GPU w/Q_K MoE quants on CPU which is some may find more satisfying than FP8->Q8_0 conversion.

And also keep an eye on the new --numa mirror PR #16000 .