Race with multus daemonset when autoscaling

[netiperher]

This approach described here with lamda that assigns extra ENI and the "Automated Multus pod IP management" init container works well for me in order to get extra interface in pod with multus. In my setup primary pod interface is still managed by aws-vpc-cni.

My only issue is that when adding more instances (due to cluster-autoscaler) is that sometimes my workload gets started before the multus dameonset has finished running. When this happens only the /etc/cni/net.d/10-aws.conflist exists and not the /etc/cni/net.d/00-multus.conf causing pod to start without extra interfaces and the "Automated Multus pod IP management" container gets stuck forever.

Is there any known fix for this?

[raghs-aws]

One of the option could be adding the container as a sidecar instead of initcontainer. This should add the secondary IPs when new interfaces get added. Havent tested though this case.

Did the pod not go in wait state as it would on the multus interfaces. if it doesn't then I believe its not handled in initcontainer, as it assumes pod networking is complete. we can plan to have a MAX delay (sleep) to handle such cases. you can try something like this in your initiContainer invocation
args: [/bin/sh, -c, 'sleep 30; /app/script.sh initContainers']
Once again havent tested this code.

[netiperher]

My understanding is that the issue is with the kubelet's logic. It runs the cni interface creation logic i.e. for both aws-cni and multus when pod is initialized. This happens before any (init)containers are started. So if only the 10-aws.conflist exists on host when pod is started then the multus part is skipped (since kubelet doesn't know about it yet).

So I don't think adding any sleep or switching to sidecar would help here since the actual network interface inside the pod doesn't exist and will never be created (later) for this pod.

It might be possible to have some timeout in the initcontainer and then fail container after some time, but just restarting the container is not enough though since cni creation is on pod level. So possible this needs to be combined with a restartPolicy: Never on the pod and then having a controller re-create the actual pod.

[raghs-aws]

makes sense, unless pod is recreated, it will not add secondary interfaces... somehow I havent seen this issue... I believed multus being master CNI would be added first, but its possible that its not complete and some race condition happens. Is it possible that your pods are allocated to node when its not done initializing or not ready.
another way could be delay the node ready via ec2 user-data. i.e. add a kubelet stop/start. I know these are some workarounds and not a solution.

we need to investigate more on this. you could also raise a support ticket on EKS to see if there are any known solutions.

[netiperher]

I think one factor is that I'm deploying multus with this image 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/multus-cni:v4.0.2-eksbuild.1_thick from an EKS cluster in Europe, so that could perhaps explain why it is slower for me.

I tested a workaround now to bootstrap the multus config from launch template. The consequence then is that pod fails to initialise (failed to find plugin "multus-shim" in path) but kubelet will periodically retry so eventually pod gets created.

      userData: ec2.UserData.custom(
        `MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
ls /sys/class/net/ > /tmp/ethList;cat /tmp/ethList |while read line ; do sudo ifconfig $line up; done
grep eth /tmp/ethList |while read line ; do echo "ifconfig $line up" >> /etc/rc.d/rc.local; done
systemctl enable rc-local

cat <<EOF > /etc/cni/net.d/00-multus.conf
{
  "capabilities": {
    "portMappings": true
  },
  "cniVersion": "0.3.1",
  "logFile": "/var/log/multus.log",
  "logLevel": "verbose",
  "name": "multus-cni-network",
  "clusterNetwork": "/host/etc/cni/net.d/10-aws.conflist",
  "type": "multus-shim",
  "socketDir": "/host/run/multus/",
  "multusMasterCNI": "10-aws.conflist"
}
EOF
--==MYBOUNDARY==--`

But feel free to let me know if you find any better solutions 😀