LocalAI always shows "Fails: could not load model: rpc error"

[philipp-fischer]

I built localai locally (not as docker), since I'm on an ARM architecture with CUDA.
I am able to run the ./localai binary and talk to the server, however as soon as I want to run inference with a model, the server doesn't seem to find a backend that works:

[...]
1:12PM INF Model name: gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin
1:12PM INF Trying to load the model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with all the available backends: llama-cpp, llama-ggml, llama, gpt4all, bert-embeddings, rwkv, whisper, stablediffusion, tinydream, piper
1:12PM INF [llama-cpp] Attempting to load
1:12PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend llama-cpp
1:12PM INF [llama-cpp] Fails: could not load model: rpc error: code = Canceled desc =
1:12PM INF [llama-ggml] Attempting to load
1:12PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend llama-ggml
1:12PM INF [llama-ggml] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
1:12PM INF [llama] Attempting to load
1:12PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend llama
1:13PM INF [llama] Fails: could not load model: rpc error: code = Canceled desc =
1:13PM INF [gpt4all] Attempting to load
1:13PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend gpt4all
1:13PM INF [gpt4all] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
1:13PM INF [bert-embeddings] Attempting to load
1:13PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend bert-embeddings
1:13PM INF [bert-embeddings] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
1:13PM INF [rwkv] Attempting to load
1:13PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend rwkv
1:13PM INF [rwkv] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
1:13PM INF [whisper] Attempting to load
1:13PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend whisper
1:13PM INF [whisper] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/whisper. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
1:13PM INF [stablediffusion] Attempting to load
1:13PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend stablediffusion
1:13PM INF [stablediffusion] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
1:13PM INF [tinydream] Attempting to load
1:13PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend tinydream
1:13PM INF [tinydream] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/tinydream. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
1:13PM INF [piper] Attempting to load
1:13PM INF Loading model 'huggingface@gorilla-llm__gorilla-7b-hf-v1-ggml__ggml-model-q5_0.bin' with backend piper
1:13PM INF [piper] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/piper. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS

Any ideas?

[philipp-fischer]

@mudler, any idea what's going wrong here? I'm sure the model is in the right ./models folder.
How do I know which is the right backend for a given model?

[mudler]

@philipp-fischer can you share the logs with the --debug flag on? it will show more output from the backends, so we can actually check what's going on there

[philipp-fischer]

Oh I think I just solved it using those logs. The file should not have the huggingface@ prefix it seems. It wasn't able to find the bin file.

[philipp-fischer]

Hm, it doesn't seem to be using the GPU at all. Just CPU.

[philipp-fischer]

Ok solved that, too, by modifying the yaml file. Thanks.

[qingfenghcy]

Hello, specifically what configuration can be adjusted to run, and what configuration has been adjusted to use gpu other than cpu.THX.

[961815748]

Hello, I also encountered this problem, can you say how did you modify it? thanks very much