Will there ever be a GPU support for Apple Silicon?

[alexcardo]

I really thank you for the possibility of running the model on my MacBook Air M1. I've been testing various parameters and I'm happy even with the 7B model. However, do you plan to utilize the GPU of M1/M2 chip? Thank you in advance.

[gjmulder]

Not any time soon. The code is written to work extremely fast on CPUs, specifically x86_64 and ARM64 systems, and not GPUs.

The M1/M2 GPUs probably do not have enough memory to load the models, but check out FB's original LLaMM inference code to run FB's inference engine and DeepSpeed for techniques to distribute models across both CPU and GPU memory. This can get very involved.