Best settings for Maverick - Dual CPU Xeon 8480+ - RTX 3090

[justinjja]

With a single 8480+ and a 3090 I get excellent speeds ~40 T/s on Maverick
After installing a second cpu and another 8 sticks of ram I cant get good speeds.
numa distribute gives ~27 T/s
numa isolate (and -t 56) is even slower at ~10 T/s
(With cache cleared between tests)

This is with Sub-NUMA Clustering disabled, so only 2 numa nodes total.

Any recommendations for settings that will get over 40 T/s?
Do I not understand what numa isolate does? I thought that would be the same as a single CPU.

llama-server -m Maverick-UD-IQ4_XS.gguf -c 32000 -fa -fmoe -amb 512 -rtr -ctk q8_0 -ctv q8_0 -ngl 99 -ot ".ffn_._exps.*=CPU" --numa isolate -t 56

[justinjja]

Small update,

I replaced --numa isolate with --numa numactl
and added: numactl --physcpubind=0-55,112-167 --membind=0 before my command

This does what I thought isolate would do.
I'm back at 40 T/s

Still no luck finding settings that actually both cpus.

[ad-astra-video]

Curious, does --membind=0 and using cores from the other processor provide speed up? I was messing with Kimi K2 on an old xeon server and I saw some speed up adding more cores with keeping the memory allocated on one node.

[ikawrakow]

There have been a lot of discussions around the Internet about llama.cpp performance on dual-socket systems, and the conclusion appears to be that the best one can do is to just use one physical CPU.

I don't have access to a dual socket system, so have done nothing related to NUMA in ik_llama.cpp. Hence, being a fork of llama.cpp, I expect it to behave the same.