Failed to Create QP

[zlwfrank]

I tried to deploy the rdma device plugin in HCA mode in my kubernetes cluster. I followed the instruction and the device plugin can be registered successfully. If I run "kubectl describe node [node_name]", I can find the rdma/hca resource. If I run "ibstat" in the pods, the inifiniband information shows up and the status is active/up.

However, when I tried to run a connection test using "ib_read_bw", it threw me following error: "Couldn't get device attribute.
Unable to create QP.
Failed to create QP.
Couldn't create IB resource."

I simply run the test by running "ib_read_bw" in one pod and running "ib_read_bw [target_pod_ip_addr]" in another pod. Could anyone please help with this issue? I appreciate your help.

[paravmellanox]

@zlwfrank
container might not have IPC_LOCK capabilities.

Refer to example here to add "IPC_LOCK" line at appropriate place.

spec:
restartPolicy: OnFailure
containers:

• image: mellanox/mlnx_ofed_linux-4.4-1.0.0.0-centos7.4
  name: mofed-test-ctr
  securityContext:
  capabilities:
  add: [ "IPC_LOCK" ]

[zlwfrank]

@paravmellanox Thanks for the reply. Actually I was using the provided sample .yaml file and the IPC_LOCK capability had been added.

This is the file I used:

apiVersion: v1
kind: Pod
metadata:
name: ib-test-pod-1
spec:
restartPolicy: OnFailure
containers:

• image: mellanox/centos_7_4_mofed_4_2_1_2_0_0_60
  name: mofed-test-ctr
  securityContext:
  capabilities:
  add: [ "IPC_LOCK" ]
  resources:
  limits:
  rdma/hca: 1
  command:
  • sh
  • -c
  • |
    ls -l /dev/infiniband /sys/class/net
    sleep 1000000

[yh-xu]

@zlwfrank have you resolved this problem? I got the same symptom of "fail to create qp" when running ib_read_bw inside container, and had no idea how to deal with.