Trying to run in a docker container on CPU throws LLAMA_ASSERT: /workspace/llama-util.h:101: ret == 0 Aborted

[michaelbauerinc]

Hello, I've been having a blast with llama-cpp, thanks so much to everyone working hard on the development of this.

I'm trying to run the binary inside of a linux docker container. It runs fine on my host machine (intel i7, no gpu) when compiled for mac os, but when trying to execute this in multiple different flavors of linux inside docker, I get this error:

root@903bba1630aa:/workspace# ./build/bin/main -m ./model_bins/llama-2-7b-chat.ggmlv3.q4_0.bin 
main: build = 919 (edcc7ae)
main: seed  = 1690657968
LLAMA_ASSERT: /workspace/llama-util.h:101: ret == 0
Aborted

I'm not sure exactly what this means or how to reconcile it. If anyone here is capable of shedding some light on this/helping me troubleshoot, it would be greatly appreciated :)

[michaelbauerinc]

I also get this same error by following the documentation in the readme for docker exactly

docker run -v $(pwd)/model_bins/:/models ghcr.io/ggerganov/llama.cpp:light -m /models/llama-2-7b-chat.ggmlv3.q8_0.bin -p "Building a website can be done in 10 simple steps:" -n 512

main: build = 1 (8a88e58)
main: seed  = 1690661944
LLAMA_ASSERT: llama-util.h:101: ret == 0

[michaelbauerinc]

Solved - the issue is the volume. If you explicitly copy the binary to the container instead of mounting it, everything works as expected.

This tracks, docker desktop with mac has deep performance issues with volumes. Going to close this thread, ty 4 coming 2 my ted talk