[question] Error while loading shared libraries: libcuda.so.1: on Pod Initialization (CrashLoop)

[jothipillay]

What happened:

Followed instruction to setup Koordinator v1.7 with Hami Core. Intended Purpose is to achieve GPU Memory Isolation and Fractioning.

When i use the sample pod specs provided. Pod goes into Crash Loop Back Off error state with following error
"error while loading shared libraries: libcuda.so.1:"

full screenshot below

Can advise what I did wrong or troubleshooting steps. Which component is this issue related to?

Environment:

• Koordinator version: - v1.7
• Kubernetes version (use kubectl version): v1.29.x
• docker/containerd version: containerd 1.7.28
• OS (e.g: cat /etc/os-release): Ubuntu 22.04
• Kernel (e.g. uname -a): 5.15.0-161-generic

Anything else we need to know:
GPUEnvInject enabled in Koordlet
Also tried with Koordinator v1.6 with Hami Core Distribute - faced same error.

[AutuSnow]

@jothipillay According to my understanding, there is a problem with the video card driver on the machine. Can you check whether the lower drive is normal？

[jothipillay]

HI @AutuSnow, thank you for the reply.

Do you mean the NVIDIA driver itself? I currently tried with 2 versions (v535.183.06) and (v580.126.09) and facing same issue with both. Is there a recommended NVIDIA driver version I should try with for Koordinator v1.7 and Koordinator v1.6

Also I think the driver is ok by itself. Without koordinator, I was able to create pods with nvidia.com/gpu resources and run workloads without issues.

The issue only comes when i add the koord-scheduler annotation "schedulerName: koord-scheduler" to the pod specs to schedule the pod through koordinator. Then the pod is not able to go into Running State.

[AutuSnow]

Can you provide Koordlet's configmap configuration and pod resource request configuration field information？

[ZiMengSheng]

HI @AutuSnow, thank you for the reply.

Do you mean the NVIDIA driver itself? I currently tried with 2 versions (v535.183.06) and (v580.126.09) and facing same issue with both. Is there a recommended NVIDIA driver version I should try with for Koordinator v1.7 and Koordinator v1.6

Also I think the driver is ok by itself. Without koordinator, I was able to create pods with nvidia.com/gpu resources and run workloads without issues.

The issue only comes when i add the koord-scheduler annotation "schedulerName: koord-scheduler" to the pod specs to schedule the pod through koordinator. Then the pod is not able to go into Running State.

Could you provide the Pod YAML?

[jothipillay]

Hi @AutuSnow @ZiMengSheng

Providing all the requested information below. Kindly let me know if any other information required.

Pod YAML

Koordlet DaemonSet Container Args

[ZiMengSheng]

@jothipillay Have you installed the nvidia-container-toolkit and configure the pod with spec.runtimeClassName=nvidia?

[jothipillay]

@ZiMengSheng yes i have the whole gpu operator running (with nvidia-container-tooklit daemonset) and runtime class name to be nvidia. this still causes the issue.

My pod works without issues when I remove "schedulerName: koord-scheduler" and request for "nvidia.com/gpu" resource instead as per normal GPU resource request flow in kubernetes.

Can I check, for this error "Error while loading shared libraries: libcuda.so.1" ===> which component is trying to load the shared libraries and throwing this error?

[AutuSnow]

@jothipillay Can you provide the information of your machine according to the instructions?
ssh poc-test-kw12
ldd /usr/local/vgpu/libvgpu.so
If present：
ldd libvgpu.so | grep cuda

[AutuSnow]

@jothipillay Can you add spec.runtimeClassName=nvidia in your pod's yaml file to try to start

[AutuSnow]

@ZiMengSheng The problem may be that koordlet injects LD_PRELOAD configuration, but when the container starts, it tries to load/usr/local/vgpu/libvgpu.so, but libvgpu.so depends on libcuda. so.1

[ZiMengSheng]

@jothipillay Can you provide the information of your machine according to the instructions? ssh poc-test-kw12 ldd /usr/local/vgpu/libvgpu.so If present： ldd libvgpu.so | grep cuda

@AutuSnow Yes, you are right, this check is necessary. Could you please help check the dependencies of HAMi-Core using the commands above? @jothipillay

[jothipillay]

Hi @AutuSnow @ZiMengSheng

Sorry for the late reply. Was away from my test environment for awhile.

I tried adding the spec.runtimeClassName=nvidia to pod yaml and face the exact same error.

With regards to information on machine, here is the info requested

Could it be an associated issue with NRI or not related?

[AutuSnow]

/hold Wait