Adds support for 4bit (nf4) and 8bit bitsandbytes quantization (3/3)#151

Open

Rypo wants to merge 15 commits intoVectorSpaceLab:mainfrom

Rypo:bnb_quantization

Commits on Nov 26, 2024

Commits on Dec 2, 2024

Commits on Dec 3, 2024

feat: support quantization with prequantized weights, add auto detection for bnb quant dict
Rypo
committed
feat: support loading prequantized weights from the hub
Rypo
committed

Commits on Dec 6, 2024

Merge branch 'main' into bnb_quantization
Rypo
committed

Commits on Dec 12, 2024

fix: prevent float16 numerical overflow
Rypo
committed

Commits on Dec 17, 2024