Problems with EKS and VPC-CNI #4063

jabbrwcky · 2023-02-13T11:54:09Z

jabbrwcky
Feb 13, 2023

Hi, I am currently trying to deploy an EKS cluster with VPC-CNI (capa v2.0.2) with addons (coredns and kube-proxy) using the
template eks-managedmachinepool-vpccni.

clusterctl generate cluster infra --flavor eks-managedmachinepool-vpccni --kubernetes-version v1.23.0 --worker-machine-count=3 --infrastructure aws > eks-infra.yaml

The coredns addon fails to deploy because of a failure to assign an ip address: network: add cmd: failed to assign an IP address to container

Anybody can give me a hint, what I am missing in my configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: infra
  namespace: aws-infrastructure
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: infra-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSManagedCluster
    name: infra
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedCluster
metadata:
  name: infra
  namespace: aws-infrastructure
spec: {}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: infra-control-plane
  namespace: aws-infrastructure
spec:
  eksClusterName: infrastructure
  # network:
  #   cni:
  #     cniIngressRules: []
  sshKeyName: default
  vpcCni:
    disable: false
    env:
    - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
      value: "true" 
    - name: ENABLE_PREFIX_DELEGATION
      value: "true"
    - name: MINIMUM_IP_TARGET
      value: "32"
    - name: "WARM_IP_TARGET"
      value: "6"
  addons:
  - name: vpc-cni
    version: v1.12.1-eksbuild.2
  - name: kube-proxy
    version: v1.23.15-eksbuild.1
  - name: coredns
    version: v1.8.7-eksbuild.3
  region: eu-central-1
  version: v1.23.0
  associateOIDCProvider: true
  endpointAccess:
    public: true
    private: true

Skarlso · 2023-02-13T11:59:02Z

Skarlso
Feb 13, 2023

Hi 👋

Can you try it with this?

    - name: "vpc-cni"
      version: "v1.12.1-eksbuild.2"
      conflictResolution: "overwrite"

5 replies

jabbrwcky Feb 13, 2023
Author

I'll try, but I had the conflictResolution set to 'overwrite' IIRC.

It looks like the VPC CNI is expecting an ENIConfig resource named default that is missing:

{"level":"info","ts":"2023-02-13T13:10:48.410Z","caller":"ipamd/ipamd.go:837","msg":"Found ENI Config Name: default"}
{"level":"error","ts":"2023-02-13T13:10:48.410Z","caller":"ipamd/ipamd.go:837","msg":"error while retrieving eniconfig: ENIConfig.crd.k8s.amazonaws.com \"default\" not found"}
{"level":"error","ts":"2023-02-13T13:10:48.410Z","caller":"ipamd/ipamd.go:811","msg":"Failed to get pod ENI config"}

From the capa logs it looks like some ENIConfig is created:

capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:02.904773       1 logger.go:68] "Reconciling AWSManagedControlPlane"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:03.327962       1 logger.go:68] "Reconciling subnets"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:03.876290       1 logger.go:68] "using eks control plane role" role-name="infra-iam-service-role"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:04.254751       1 logger.go:68] "Reconciling EKS security groups" cluster-name=0xc00142abd0
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:04.361479       1 logger.go:68] "reconciling encryption configuration"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:04.361512       1 logger.go:68] "Reconciling EKS addons"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.441921       1 logger.go:68] "reconciling oidc identity provider"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.441945       1 logger.go:68] "no oidc provider config, skipping reconcile"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.441958       1 logger.go:68] "Reconciling aws-node DaemonSet in cluster" cluster="aws-infrastructure/infra"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.583206       1 logger.go:68] "updating aws-node daemonset environment variables" cluster="aws-infrastructure/infra"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.583259       1 logger.go:68] "for each subnet" cluster="aws-infrastructure/infra"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.593556       1 logger.go:68] "Updating ENIConfig" cluster="aws-infrastructure/infra" subnet="subnet-098c2293b195ebf98" availability-zone="eu-central-1a"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.617567       1 logger.go:68] "Updating ENIConfig" cluster="aws-infrastructure/infra" subnet="subnet-0b6d3873d3bb8dea7" availability-zone="eu-central-1b"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.642621       1 logger.go:68] "Updating ENIConfig" cluster="aws-infrastructure/infra" subnet="subnet-0309fc6366a6d7861" availability-zone="eu-central-1c"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.665106       1 logger.go:68] "updating containers" cluster-name="infra" cluster-namespace="aws-infrastructure"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.685274       1 logger.go:68] "Reconciling kube-proxy DaemonSet in cluster" cluster="aws-infrastructure/infra"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.716162       1 logger.go:68] "Reconciling aws-iam-authenticator configuration" cluster="aws-infrastructure/infra"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:09:05.868559       1 logger.go:68] "Reconciled aws-iam-authenticator configuration" cluster="infra"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:12:20.522796       1 logger.go:68] "identity: Reconciling AWSManagedMachinePool" identityKey="/default"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:12:20.523022       1 logger.go:68] "identity: Reconciling AWSManagedMachinePool" identityKey="/default"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:12:20.523128       1 logger.go:68] "identity: Reconciling AWSManagedMachinePool" identityKey="/default"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:12:21.363739       1 logger.go:68] "identity: Reconciling ASG tags" identityKey="/default" cluster-name="infrastructure" nodegroup-name="aws-infrastructure_infra-pool-1c01"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:12:21.562234       1 logger.go:68] "identity: Reconciling ASG tags" identityKey="/default" cluster-name="infrastructure" nodegroup-name="aws-infrastructure_infra-pool-1b01"
capa-controller-manager-884599ddf-mbnkd manager I0213 13:12:21.616922       1 logger.go:68] "identity: Reconciling ASG tags" identityKey="/default" cluster-name="infrastructure" nodegroup-name="aws-infrastructure_infra-pool-1a01"

jabbrwcky Feb 13, 2023
Author

I found that CAPA actually creates three ENIConfigs, one per AZ/Subnet:

kubectl get ENIConfig -A
NAME            AGE
eu-central-1a   7m12s
eu-central-1b   7m12s
eu-central-1c   7m12s

The aws-node seems to look for a default ENIConfig 🤔

jabbrwcky Feb 13, 2023
Author

Ok, I think I figured it out.

I use three nodegroups (one per AZ) to enable using persistent volumes (and autoscaling eventually). So CAPA (correctly) builds one ENIConfig per AZ, named after the AZ.

I assume that if you create a cluster with a nodegroup spanning several AZs an ENIConfig named default will be created (and expected by VPCCNI).

What was missing to make it work in my case was:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
# ...
spec:
  vpcCni:
  # ...
    env:
    - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
      value: "true" 
    - name: ENI_CONFIG_LABEL_DEF
      value: failure-domain.beta.kubernetes.io/zone

My full working configuration looks like this now (I think conflictResolution: overwrite is not strictly necessary, because it is the default for the current CRD Version):

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedCluster
metadata:
  name: infra
  namespace: aws-infrastructure
spec: {}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: infra-control-plane
  namespace: aws-infrastructure
spec:
  eksClusterName: infrastructure
  # network:
  #   cni:
  #     cniIngressRules: []
  sshKeyName: default
  secondaryCidrBlock: 100.64.0.0/16
  vpcCni:
    disable: false
    env:
    - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
      value: "true" 
    - name: ENABLE_PREFIX_DELEGATION
      value: "true"
    - name: MINIMUM_IP_TARGET
      value: "32"
    - name: "WARM_IP_TARGET"
      value: "6"
    - name: ENI_CONFIG_LABEL_DEF
      value: failure-domain.beta.kubernetes.io/zone
  addons:
  - name: vpc-cni
    version: v1.12.1-eksbuild.2
    conflictResolution: "overwrite"
  - name: kube-proxy
    version: v1.23.15-eksbuild.1
    conflictResolution: "overwrite"
  - name: coredns
    version: v1.8.7-eksbuild.3
    conflictResolution: "overwrite"
  region: eu-central-1
  version: v1.23.0
  associateOIDCProvider: true
  endpointAccess:
    public: true
    private: true

Skarlso Feb 13, 2023

Ah interesting. Thanks. I think this warrants a documentation update maybe. :)

Ankitasw Feb 20, 2023
Maintainer

Do we have an issue in place for this?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Problems with EKS and VPC-CNI #4063

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Problems with EKS and VPC-CNI #4063

Uh oh!

jabbrwcky Feb 13, 2023

Replies: 1 comment · 5 replies

Uh oh!

Skarlso Feb 13, 2023

Uh oh!

jabbrwcky Feb 13, 2023 Author

Uh oh!

jabbrwcky Feb 13, 2023 Author

Uh oh!

jabbrwcky Feb 13, 2023 Author

Uh oh!

Skarlso Feb 13, 2023

Uh oh!

Ankitasw Feb 20, 2023 Maintainer

jabbrwcky
Feb 13, 2023

Replies: 1 comment 5 replies

Skarlso
Feb 13, 2023

jabbrwcky Feb 13, 2023
Author

jabbrwcky Feb 13, 2023
Author

jabbrwcky Feb 13, 2023
Author

Ankitasw Feb 20, 2023
Maintainer