Please release a mini version with low VRAM!

[dream80]

The current model cannot be used by ordinary people at all. We need a model with smaller parameters and good capabilities. A good model is one that everyone can play with.

Anyone have the same idea?

[bluevisor]

I'm sure there are people who know what they are doing already working on this.

[StavrosD]

This is a HUDGE model, about 300B parameters. With 8bit quantization you need about 300GB VRAM.
Even if you quantize it to 1bit (worst quality) you need 100GB VRAM, you need 5 GPUs with 24GB VRAM each.
If we are lucky we will see a quantized GGUF version. GGUF supports loading to RAM and offloading some layers to the VRAM.

[AdaptiveStep]

You could run this as an entire game engine, and let just its context size be responsible for the entire game state lol.

[nukedsystem]

Is it a model for millionaires or billionaires? Is it an Elite model?