I wonder if we can use GPTQ quantized models for this? 7B HF models are too big even for 12Gb cards...

[barleyj21]

7B HF models take more than 12Gb of memory, so is there a way to use GPTQ ones?

[PromtEngineer]

Its a good idea. I am testing them and also looking at adding support for different embeddings.

[PromtEngineer]

Just saw the new accelerate update. I think it will now be possible.

[LitenBuzzTh]

Support for GPTQ models would be awesome

[KonradHoeffner]

You can use GPTQ now.

[LitenBuzzTh]

yeah been using it for a while