What is the chance of running Llama langauge model locally?

[elephantpanda]

Hi with the new Llama language model from Meta. What is the possibility that one could run this successfully locally on, say a GeForce 3080 32GB GPU using Onnxruntime.

Even though Llama is a rival to Chat GPT from Microsoft(!!)

It is reported to have a 7B parameter model. Which I guess in float16 would come to 14GB.

Should I even be asking this question 😯. Well even though it is from Meta people still want to run it on Microsoft Windows....

[elephantpanda]

People have got the model to run locally on GPU with 16GB VRAM. (CUDA) using torch. Maybe someone from Onnxruntime should try it to make sure it works with your system too.

[ivanstepanovftw]

See https://github.com/ggerganov/llama.cpp, maybe they already have WASM module for it. They also have managed to run quantized LLAMA on Android in Termux.

[elephantpanda]

See https://github.com/ggerganov/llama.cpp, maybe they already have WASM module for it. They also have managed to run quantized LLAMA on Android in Termux.

Thanks. I actually got the 4bit version running as well. But for certain situations it would be nice to run this as an onnx file. Just because it's easier to integrate into a c# Windows program.

[ivanstepanovftw]

@pauldog, do you occasianally know how to use linear algebra using ONNX.js, ONNX Runtime or some other library? I have opened discussion, any suggestion may help.