k8s场景下 cuda报错

[wwssttcc]

你好，我在k8s场景下搭建orion，客户端运行cuda例子，报以下错误：
root@orion-client:~/cuda10.0-regression/cuda_samples/vectorAdd# ./vectorAdd
./vectorAdd: /usr/lib/orion/libcudart.so.10.0: no version information available (required by ./vectorAdd)
[Vector addition of 50000 elements]
VirtaiTech Resource. Build-cuda-bd1163c-20200128_130008
2023-07-28 07:08:13 [INFO] Using Orion resource (439abbd1-51da-4a13-81cf-4b77d8e728a4) f8973b72-7967-40c4-8eda-575e60234d0d : 172.17.0.1:9960/0/0
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
2023-07-28 07:08:14 [ERROR] Failed to load fatbinary image into device, cuda errno: 221, please use Ctrl + C to shutdown application
2023-07-28 07:08:14 [ERROR] Failed to load kernel _Z9vectorAddPKfS0_Pfi, cuda errno: 400, please use Ctrl + C to shutdown application
Copy output data from the CUDA device to the host memory
Result verification failed at element 0!
2023-07-28 07:08:14 [INFO] Releasing Orion resource ...

从上面可以看出是cuda errno: 221报错了，请问下这是什么原因？