When are 1bit quants coming?

[superchargez]

So, there is a paper The Era of 1-bit LLMs:
All Large Language Models are in 1.58 Bits by same people who wrote BitNet: Scaling 1-bit Transformers for
Large Language Models in October last year. So, is anybody going to try creating a 1bit quant algorithms?

[WiseFarAI]

It's already here! See: #5453

[sorasoras]

That's not a quant

[superchargez]

I don't know what to call it, it is not 2bit either, trinary bit maybe? but does it equate to 1.58? i don't think it should be called 1.58 bit either, how does it even calculate. maybe @sorasoras is right current CPUs don't efficiently make this calculation either that is why it is slow, but if they could I think they should make it 70 times faster than fp16, what you think?

[JJones780]

If you look at #5453 there is some discussion of separating salient ( +/-1 ?) and non-salient ( i.e. 0 ?) weights - which I'm guessing is how you get 1.5 with a bit extra for the overhead of specifying the boundary etc. It's not a name per se but an actual approximation of the average size of the weights. I assumed Trinaries represented in binary would require 2 bits without compression of some type... and wasting a state. It had me wondering too.

ALN's aimed for real-time application back when computers ran at 300MHz. They could be implemented in hardware by fpgas etc.

Speaking of hardware.. don't quantum computers have 3 states...

[iplayfast]

if it's trinary then it wouldn't take any extra space to make it quadary. Instead of 1 -1 and 0, we would have 1, -1, 0.001 and -0.001
There would be no zero. I think that would make it denser.

[superchargez]

but that is (would be) just normalization, will it affect anything, like speed or space taken up by model? I think it would get worsen because 0 is easier to calculate and store than non zero.

[jukofyork]

I don't know what to call it, it is not 2bit either, trinary bit maybe? but does it equate to 1.58? i don't think it should be called 1.58 bit either, how does it even calculate.

The 1.58 number comes from log2(3).

The term "bit" here isn't quite what most people are used to and actually relates to Information Theory:

https://en.wikipedia.org/wiki/Information_content

Specifically, when you use log2 to measure the information content it's now known as a "bit" but at one time it was more commonly known as a "Shannon Unit":

https://en.wikipedia.org/wiki/Shannon_%28unit%29

If you imagine a wire that let's you send only on/off signals and the on and off signals have uniform 50/50 chance of occurring, then it should be clear the information content of a single signal is 1 bit.

If you now have a wire that let's you send 3 different levels of signal and each has a 1/3 chance of occurring, the amount of information you can send in 1 signal is clearly more... The exact value compared to the on/off only wire turns out to be log2(3).

You can extend this to any number of levels, but the important part is they must have equal chance of occurring (aka "Maximum Entropy") or else you could send more using compression...

If you are more interested:

https://jamesstone.sites.sheffield.ac.uk/books/information-theory

or:

https://mirtitles.org/2022/03/30/probability-and-information-yaglom-yaglom/

are good introductions without being too mathematical.

[iplayfast]

Is this the same as https://www.youtube.com/watch?v=Gtf3CxIRiPk&ab_channel=1littlecoder

[Kreijstal]

0 bits quantization is just rm -rf

[iplayfast]

https://arstechnica.com/information-technology/2024/06/researchers-upend-ai-status-quo-by-eliminating-matrix-multiplication-in-llms/