-
|
I'm trying to do a robust quantization over 34 languages, so I have a rather large dataset for the llama-imatrix file. On my m3 Max macbook, this will take >260hrs, and I've had some crashes it seems every time I wake the screensaver, so I decided to try using vast.ai hosting. I am using the ghcr.io/ggerganov/llama.cpp:full-cuda image from this repo. Something definitely is amiss though, On my macbook m3 max, the ETA to complete from llama-imatrix is ~260hrs The first time I tries, the system I chose was an AMD Epic with 4x4090. the ETA was 660 hrs. I was expecting ~8 fold improvement, not ~3 fold slowdown. I used a script to execute the command though, and it didn't record the commands used, I worried it might not have captured the number of gpus correctly and ended up passing bad params. So, I tried again. This time on a Xeon server with 4x4090s. This ETA was 636 hours, sorry I cut it off on the version of the log that I downloaded. Why is it taking almost 3x as long to do this on a 4x4090 machine as it is locally? |
Beta Was this translation helpful? Give feedback.
Answered by
slaren
Oct 17, 2024
Replies: 1 comment 5 replies
-
|
The CUDA backend also does not support BF16, so most of the model is running on the CPU. Try a F16 model instead. Also note that |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
ahaha when you said "keep in mind that BF16 is not supported in the Metal backend" last time I thought that it was unique to Metal :D Damn, but I do want to do this in native bf16 since the original is. Is there no acceleration available to me? (even still, my Mac laptop outperforming a Xeon 3x is still kinda surprising) |
Beta Was this translation helpful? Give feedback.
-
|
F16 is just as good in essentially every case. Most models are normalized so that the weights have a distribution of |
Beta Was this translation helpful? Give feedback.
-
|
I .. thank you. can you give me a reference regarding that latter claim? I.. the purpose to creating these imatrices is so when I quantize it, I minimize ppl loss (over the target languages). You're saying, if I understand this correctly, I could go straight to Q8_0 with no imatrix, generate the imatrix from that, then go back and use that imatrix to recreate q8 and all the stronger quants? |
Beta Was this translation helpful? Give feedback.
-
|
See for example #5222. Admittedly, the results may be different with different models, but typically |
Beta Was this translation helpful? Give feedback.
-
|
wow this has been such a treasure of a discussion, thank you @slaren |
Beta Was this translation helpful? Give feedback.
Answer selected by
robbiemu
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The CUDA backend also does not support BF16, so most of the model is running on the CPU. Try a F16 model instead.
Also note that
-ngl -1does not work the way you might expect, no layers will be offloaded that way. Use a large number to offload the entire model instead, eg.-ngl 99.