Replies: 9 comments 19 replies
-
|
Anyone who has the horse power to run Qwen3-235B-A22B, please feel free to add your results to this discussion. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
I'm away from home but frantically trying to remote into a server I just got access too again and cook up a good Qwen3-235B-A22B mix for my home 3090TI 24GB VRAM + 96GB RAM system which is about the limit of common AM5 gaming rigs (with the faster and more supported 2x DIMM configuration). Any particular reason you chose I haven't finished yet but my very rough WIP custom quantize script so far is: Very rough ik_llama.cpp custom quantize script#!/usr/bin/env bash
custom="
#token_embd.weight - [ 4096, 151936, 1, 1], type = bf16, Using custom type q8_0 for tensor token_embd.weight
#blk.1.ffn_gate_inp.weight - [ 4096, 128, 1, 1], type = f32, size = 2.000 MB
#blk.1.attn_k_norm.weight - [ 128, 1, 1, 1], type = f32, size = 0.000 MB
#blk.1.attn_q_norm.weight - [ 128, 1, 1, 1], type = f32, size = 0.000 MB
#blk.1.attn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB
#blk.1.ffn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB
#blk.1.attn_k.weight - [ 4096, 512, 1, 1], type = bf16, Using custom type q8_0 for tensor blk.1.attn_k.weight
#blk.1.attn_q.weight - [ 4096, 8192, 1, 1], type = bf16, Using custom type q8_0 for tensor blk.1.attn_q.weight
#blk.1.attn_v.weight - [ 4096, 512, 1, 1], type = bf16, Using custom type q8_0 for tensor blk.1.attn_v.weight
#blk.1.attn_output.weight - [ 8192, 4096, 1, 1], type = bf16, Using custom type q8_0 for tensor blk.1.attn_output.weight
#blk.1.ffn_down_exps.weight - [ 1536, 4096, 128, 1], type = bf16, Using custom type q8_0 for tensor blk.1.ffn_down_exps.weight
#blk.1.ffn_gate_exps.weight - [ 4096, 1536, 128, 1], type = bf16, Using custom type q8_0 for tensor blk.1.ffn_gate_exps.weight
#blk.1.ffn_up_exps.weight - [ 4096, 1536, 128, 1], type = bf16, Using custom type q8_0 for tensor blk.1.ffn_up_exps.weight
#output_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB
# Token embedding
token_embd\.weight=q8_0
# Attention
blk\..*\.attn_k.*=iq6_k
blk\..*\.attn_q.*=iq4_k
blk\..*\.attn_v.*=iq6_k
blk\..*\.attn_output.*=iq4_k
# Experts
blk\..*\.ffn_down_exps\.weight=iq4_k
blk\..*\.ffn_(gate|up)_exps\.weight=iq3_k
"
custom=$(
echo "$custom" | grep -v '^#' | \
sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)
#--token-embedding-type q8_0 \
#--output-tensor-type q8_0 \
./build/bin/llama-quantize \
--custom-q "$custom" \
--imatrix /mnt/raid/models/ubergarm/Qwen3-235B-A22B-GGUF/imatrix-Qwen3-235B-A22B.dat \
/mnt/raid/models/Qwen/Qwen3-235B-A22B/Qwen3-235B-A22B-BF16-00001-of-00011.gguf \
/mnt/raid/models/ubergarm/Qwen3-235B-A22B-GGUF/Qwen3-235B-A22B-mix-IQ3_K.gguf \
IQ3_K \
24Did you bother to make an imatrix for your quant, and if so, were you able to activate enough experts with your imatrix corpus text? Thanks again, exciting times with Qwen3 MoE out and wondering if R2 is around the corner haha... |
Beta Was this translation helpful? Give feedback.
-
I wanted to have a quantized model that I can run with I'm playing with some quantization recipes for Qwen3-30B-A3B. I'll post the results tomorrow, maybe that can be useful to you for "cooking" the Qwen3-235B-A22B quants. |
Beta Was this translation helpful? Give feedback.
-
|
I run Qwen3-235B-A22B on my pc(#385 ), but the performance not better, might the memory performance of RAM is too slow... |
Beta Was this translation helpful? Give feedback.
-
|
Just "cooked" my first
Looks pretty good! Only other somewhat comparable benchmark I've seen is from latest ktransformers v0.3 on a rig with better GPU and more RAM. 👈 Logs
In the mean time, I ran a quick comparison of the Q8_0 on the remote threadripper pro 24 core using a single RTX A6000 48GB VRAM GPU and offloading the rest to CPU for a somewhat similar "hybrid inference" test. Note that for some reason
👈 Logsik_llama.cpp
llama.cpp
Interestingly I could hear my fans spin up and down periodically every 15 seconds or so as the CPU ramped up and the GPU dropped down a bit. I noticed this more on the Q8_0 test visually with |
Beta Was this translation helpful? Give feedback.
2 replies
-
This is because the |
Beta Was this translation helpful? Give feedback.
-
|
I don't have access to enough RAM+VRAM currently to run the full 👈 PPL and KLD comparisons on two test corpi
👈 ubergarm-kld-test-corpus.txtI created a ~1.6MiB plain text test corpus using The formatting is a little odd as there are no paragraphs and i used I'd rather not release it publicly easily accessible in full for various reasons, but contact me if you are doing some research or want exact comparisons with my quants (or I could possibly run your quant if I have time). Here is a snippet so you can see what it looks like: |
Beta Was this translation helpful? Give feedback.
-
|
@ubergarm Can you try the attached |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I compared your bartowski/THUDM_GLM-Z1-32B-0414-IQ4_XS.gguf GQA FA
Running the same and comparing against this previous data. Qwen3-235B-A22B-Q8_0 GQA FA
Running the same and comparing against the above chart. ConclusionThe general trends seem to hold, but your implementation seems a bit more consistent without the occasional dips unless that was just some noise or me doing something else on the machine. I'll use your adaptation going forward just to keep it as similar as possible with your comparisons. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
OK, after thinking more about this, I can see why mainline has a better large context TG performance on CUDA for Qwen3-235B-A22B (and previously noted for LLaMA-4): these models have a quite large GQA factor, and I'm still using the old CUDA FA implementation that did not take advantage of that. Improved GQA FA performance was added in this mainline PR.
|
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Interesting, yes, I first noticed this with GLM-4 (which uses GQA) in the CUDA + Flash Attention case benchmark. I still have the dream of converting an existing GQA architecture model to MLA but the additional fine-tuning required even with a fraction of the original training data seems daunting:
But until MLA catches on more across other models, it might make sense to revisit the CUDA FA implementation for GQA, if that is something that interests you. Of course as soon as R2 comes around, this fickle world will jump on the next hype train lmao... In the mean-time I'll re-run a couple Thanks! |
Beta Was this translation helpful? Give feedback.
-
But MLA is not all roses either. It took quite a bit of experimentation to arrive at a meaningful compromise between TG and PP performance. Mainline has a long way to go there (see #354). And then we have this much smaller KV cache, but then we need giant compute buffers to get meaningful performance, so we need to compute self attention in chunks to keep compute memory usage at a reasonable level, so suddenly the compute graph building becomes this huge pile of complications instead of being just a few tens of lines of simple code as ii is for the other models. And then seeing the massive drop in performance with large contexts in your DeepSeek-V3/R1 benchmarks, my guess is that it is still far from optimum. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, @artus-dev and @ubergarm asked me to run some sweeps for Qwen3-235B-A22B. My homelab has a substantial server with a VM in it that has the following allocation: I've run four sweeps as follows: Both ik_llama.cpp and llama.cpp were compiled with CUDA and OpenBLAS support. The sweeps were run with the following quants: The llama.cpp tests were conducted with the For the GPU tests, I kept the layer offloads identical between the two. This means that were was slightly less GPU VRAM utilization for the llama.cpp test because the model is smaller, but I felt that was the best way to keep the tests as comparable as I could manage: Logs for the runs are as follows: ik_llama.cpp CPU logsik_llama.cpp GPU logsllama.cpp CPU logsllama.cpp GPU logsI used the CPU performance PP comparison: GPU performance PP comparison: CPU performance TG comparison: GPU performance TG comparison: |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
One more test, I disabled pipeline parallelism (setting it to 1) and re-built ik_llama.cpp: This let me use my second 3090 and offload a little more. ik_llama.cpp 2x GPU logsllama.cpp 2x GPU logsand these are the updated GPU graphs:
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you for these results! I think it would be better to disable BLAS for both. CPU Prompt processing with Prompt processing speed on CUDA will also benefit from larger u-batches (e.g., The CUDA TG results are somewhat surprising (sharp performance drop with context length for |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
I just re-ran the above with 2x GPU for llama.cpp as well and edited the comment / graph. I was already re-running ik_llama w/o BLAS, I'll have the results of that shortly. |
Beta Was this translation helpful? Give feedback.
-
|
Posted! |
Beta Was this translation helpful? Give feedback.
-
|
Some more data, this time compiled w/ no BLAS: Note: the ik_llama.cpp CPU NO BLAS did hit a CUDA error on the very last iteration. ik_llama.cpp CPU NO BLAS logsik_llama.cpp 2x GPU NO BLAS logsik_llama.cpp BLAS vs NO BLAS PP comparison: ik_llama.cpp BLAS vs NO BLAS TG comparison: |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Oh, for CPU-only inference you want to build without CUDA. The almighty |
Beta Was this translation helpful? Give feedback.
-
|
D'oh, okay. I can redo it :) |
Beta Was this translation helpful? Give feedback.
-
|
ik_llama.cpp, no cuda, no blas: ik_llama.cpp CPU NO CUDA NO BLAS logs
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thanks! So, CPU PP is much better now and more inline with what I would have expected. Looking at the TG graph, it is clear that I still need to work on improving how the work is divided between the threads. The Qwen3 MoE models have a high GQA factor, so one should be able to achieve ~70-80% of zero-context performance at 16k tokens. But I see that the Epyc 9355 has 32 cores, so we are using hyper-threading? |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
On bare metal one achieves the best performance by setting the number of threads to the physical core count. But I have no idea how a VM will behave. You can try |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I think it's about a ~10% performance loss because it's in a VM. The system is a hypervisor though and used for other homelab things, so I'm fine taking that loss. I was able to run For completeness sake, I've disabled SMT on the host: and verified the core count with I also turned off the other three VMs on the system, so this is the "max performance" configuration that I can achieve without moving everything from the VM to the hypervisor host directly. ik_llama.cpp CPU NO CUDA NO BLAS NO SMTlogs
|
Beta Was this translation helpful? Give feedback.
-
|
So, ~30% better for PP, but not much difference for TG. I need to understand the cause of the sharp drop in TG performance for the first ~2k tokens. I'll investigate. Thanks a lot for these benchmarks! |
Beta Was this translation helpful? Give feedback.
-
|
You're welcome, let me know if you want me to re-run any of these benchmarks at some point in the future and I can pull / rebuild / re-test. Excited to see what shakes out! |
Beta Was this translation helpful? Give feedback.
-
hey, no idea what hypervisor you're running but unless it's ESXi big chances it can't handle AMD multi CCD chips or numa systems in general unless you pin ONE CCD to your VM performance is likely taking a hit |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
The Qwen3 models were officially released, and support was added in
ik_llama.cppin PR #355, so I was curious to run some performance benchmarks. As much as I would like to try the flagship model, I don't have enough horse power for that, so I experimented with Qwen3-30B-A3B, the 30B total, 3B active parameter MoE model.This time I'm using a custom quantization where all experts are quantized with
IQ4_XS, all attention tensors withQ5_K, and the output tensor isQ6_K. PPL for this model is only 1.25% above the PPL of thebf16model, so it is a pretty decent quality quantization. Benchmarks are run on a Ryzen-7950X system with an RTX-4080 GPU. Compared are the latestik_kllama.cppandllama.cppversions as of this morning (April 29 2025).CPU-only performance
The command line for
ik_llama.cppisllama.cppis similar, except that there is no-rtr -fmoe. I'm also including mainline results without Flash Attention (FA). In this case the K-cache is quantized withQ8_0and the V-cache isfp16.The following graph shows TG performance as a function of
N_KV, the number of tokens in the KV cache. Performance is pretty close for empty KV cache, with a performance gap increasing withN_KV. At 16k tokensik_llama.cppis 44% faster than mainline without FA, and 3.3 times faster than mainline with FA enabled.The next graph shows prompt processing (PP) speed as a function of
N_KV. As usual for CPU only inference,ik_llama.cppis much faster than mainline for PP - 3.3X for smallN_KV, increasing to 3.9X at 16k tokens. This is compared to mainline without FA. Compared tollama.cppwith FA enabled,ik_llama.cppis 11.2X faster.llama.cpp CPU-only performance data without FA
llama.cpp CPU-only performance data with FA enabled
ik_llama.cpp CPU-only performance data
Hybrid inference
The custom
IQ4_XSmodel is 15.4 GiB, so cannot be fully loaded on my 16 GB RTX-4080 GPU. This gives me the opportunity to try hybrid GPU+CPU inference via tensor overrides on both systems. The command line used in this case isI.e., everything is offloaded to the GPU except for the last 14 layers of the experts tensors. This leaves enough free VRAM to go up to a context of 32k tokens. In the case of
ik_llama.cpprun-time-repacking (for the experts left on the CPU) and fused MoE(ffn_up*X)*silu(ffn_gate*X)is enabled via-rtr -fmoe.The next graph shows TG performance as a function of
N_KV. Compared to DeepSeek, Here the performance advantage ofik_llama.cppis smaller and decreases with increasingN_KV. As there is no MLA involved, and we are dealing just with a standard attention mechanism, the CUDA FA improvements in this mainline PR that I have not (yet) ported over toik_llama.cppcounteract the performance gains from the fused MoE operations inik_llama.cpp, so we end up with a relatively close TG performance.The next graph shows PP performance as a function of
N_KV. Also here the performance gap decreases withN_KV, from about 60% for smallN_KV, to about 18% at 32k tokens.llama.cpp hybrid GPU+CPU performance data
ik_llama.cpp hybrid GPU+CPU performance data
Beta Was this translation helpful? Give feedback.
All reactions