Cannot allocate multiple devices to pod

[Lakshyagour]

I was working with afxdp-plugins-for-kubernetes
For my use case i needed to attach 2 device to the pod.

I used the following deploy

apiVersion: v1
kind: Pod
metadata: name: afxdp-pod
annotations:
k8s.v1.cni.cncf.io/networks: cndp-cni-afxdp0 spec:
containers:
- name: afxdp1
image: ubuntu:latest
command: ["tail", "-f", "/dev/null"]
resources:
requests:
afxdp/mypool: '2'
limits:
afxdp/mypool: '2'

I expected that 2 devices will available for use in the pod. But inside the container, I could only see one device.
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       13s   default-scheduler  Successfully assigned default/afxdp-pod to turing-04
  Normal  AddedInterface  10s   multus             Add eth0 [10.244.0.11/24] from cbr0
  Normal  AddedInterface  10s   multus             Add net1 [192.168.1.204/24] from default/cndp-cni-afxdp0
  Normal  Pulling         9s    kubelet            Pulling image "ubuntu:latest"
  Normal  Pulled          7s    kubelet            Successfully pulled image "ubuntu:latest" in 2.386145893s (2.386155184s including waiting)
  Normal  Created         7s    kubelet            Created container afxdp1
  Normal  Started         7s    kubelet            Started container afxdp1
This kubectl describe for the pod.

root@cndp-0-0:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if145: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether ce:0f:94:1a:d2:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0
6: ens259f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:9e:7b:5c brd ff:ff:ff:ff:ff:ff
    prog/xdp id 325 tag 992d9ddc835e5629 jited

Running ip link inside the container shows only one device

The afxdp-plugin logs show that 3 device were allocated and added to the container
DEBU[2023-07-17 06:37:47] [poolManager.go:233] [Allocate] Container environment variables: {
  "AFXDP_DEVICES": "ens259f1 ens259f0 enp7s0f0"
}

The bpf program is loaded on both the interface but the other device is still in the host namespace.

[garyloug]

Hi Lakshyagour,

I think your issue is line ~6 of your pod spec. You need to add a second cndp-cni-afxdp0

apiVersion: v1
kind: Pod
metadata:
name: afxdp-pod 
annotations:
k8s.v1.cni.cncf.io/networks: cndp-cni-afxdp0, cndp-cni-afxdp0 <<<<<<<<<<<<<<<<<<<<<<
spec:
containers:
- name: afxdp1
image: ubuntu:latest 
command: ["tail", "-f", "/dev/null"]
resources:
requests:
afxdp/mypool: '2' 
limits: 
afxdp/mypool: '2' 
restartPolicy: Never

The resource request down the bottom ultimately determines how many times the device plugin is called.
Unfortunately the cni is a completely different plugin and k8s api, so the number of times this is called is determined by the list of network attachment definitions provided to k8s.v1.cni.cncf.io/networks.

It's a little bit clunky, but it's just how we have to use DP and CNIs together.

KR,
Gary

[Lakshyagour]

Thank you Gary.