Problems caused by launching multiple pods at the same time

[rv64m]

Why do I get an error when I start multiple GPU-resource pods simultaneously (concurrently) using vcuda?

In vcuda loader.c, I add ferror to print errno related error message, I get it

But when I start the pods sequentially, I don't have this problem. So I guess it may be caused by a gap between the kubelet startup container and the gpu-manager placing the libcuda.so file.

[rv64m]

cc @mYmNeo

[rainfd]

As if vcuda would copy lib to container after container start up. When start multiple GPU-resource pods simultaneously, this action is not fast enough. You can try to modify your command sh -c sleep 5 && your command

[rv64m]

As if vcuda would copy lib to container after container start up. When start multiple GPU-resource pods simultaneously, this action is not fast enough. You can try to modify your command sh -c sleep 5 && your command

Oh, Thanks rainfd, I knew this solution, but I felt this way is a hat trick.

[mYmNeo]

What's the version of gpu-manager? I've fixed a problem in master branch but not released a image

[rainfd]

@mYmNeo my version is v1.0.4. What is the commit?

[mYmNeo]

@mYmNeo my version is v1.0.4. What is the commit?

tkestack/gpu-manager#130