Can't get LocalAI to run on my GPUs. Debugging info attached.

[atljoseph]

Trying to run localai with docker on my GPUs.

Uses CPU just fine.

ubuntu 22.04

docker w/ desktop:
{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime-hook",
      "runtimeArgs": []
    }
  }
}
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ sudo chown root:docker /var/run/docker.sock
$ sudo chown -R root:docker /var/run/docker

Log out and back in.

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

$ nvidia-smi
Mon Apr 29 02:06:32 2024       
| NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|   0  NVIDIA GeForce RTX 3090 Ti     Off | 00000000:01:00.0 Off |                  Off |
|  0%   28C    P8              12W / 450W |    282MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
|   1  NVIDIA GeForce RTX 3090 Ti     Off | 00000000:05:00.0  On |                  Off |
|  0%   38C    P5              32W / 450W |    467MiB / 24564MiB |     36%      Default |
|                                         |                      |                  N/A |

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
$ sudo apt-get update
$ sudo apt-get install -y nvidia-container-toolkit
$ which nvidia-container-toolkit

NVIDIA/nvidia-docker#838

$ sudo apt-get install -y nvidia-container-toolkit-base
$ nvidia-ctk --version
$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker
$ docker context use default

$ nvidia-container-toolkit --version 
NVIDIA Container Runtime Hook version 1.15.0
commit: ddeeca392c7bd8b33d0a66400b77af7a97e16cef

Still uses CPU when loading models.

$ docker run --privileged -it --rm --name local-ai \
  -p 8082:8080 \
  --gpus all \
  --runtime=nvidia \
  -e DEBUG=true \
  -e MODELS_PATH=/models \
  -e THREADS=1 \
  -v /home/joseph/LocalAI/models:/models \
  localai/localai:latest-aio-gpu-nvidia-cuda-12
  ...
  5:55AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:Einstein-v6.1-Llama3-8B-Q4_K_M.gguf ContextSize:4096 Seed:1927126212 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:1 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/Einstein-v6.1-Llama3-8B-Q4_K_M.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:}
  ...

$ docker info | grep nvidia
 Runtimes: io.containerd.runc.v2 nvidia runc

$ docker info | grep "Default Runtime"
 Default Runtime: runc

$ ls -la /dev/nv*
crw-rw-rw- 1 root root 195,   0 Apr 29 08:19 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Apr 29 08:19 /dev/nvidia1
crw-rw-rw- 1 root root 195, 255 Apr 29 08:19 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Apr 29 08:19 /dev/nvidia-modeset
crw-rw-rw- 1 root root 509,   0 Apr 29 08:19 /dev/nvidia-uvm
crw-rw-rw- 1 root root 509,   1 Apr 29 08:19 /dev/nvidia-uvm-tools
crw------- 1 root root  10, 144 Apr 29 08:19 /dev/nvram

/dev/nvidia-caps:
total 0
drwxr-xr-x  2 root root     80 Apr 29 08:19 .
drwxr-xr-x 20 root root   4920 Apr 29 08:19 ..
cr--------  1 root root 234, 1 Apr 29 08:19 nvidia-cap1
cr--r--r--  1 root root 234, 2 Apr 29 08:19 nvidia-cap2

Never uses GPU.

What else should i try?

[Aisuko]

This discussion related to #1218