Nodes Never join the EKS Cluster

[raficlosidus]

// This never worked for me, after 33 minutes . Nodes did not join, Please help

provider "aws" {
  region  = "us-east-1"
  profile = "stage"
}


# -------------------------------
# EKS Cluster Module
# -------------------------------
module "eks_al2023" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.8.0"

  name               = "stage-eks-cluster-new"
  kubernetes_version = "1.33"

  # EKS Addons
  addons = {
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {               # must apply before node creation
      before_compute = true
    }
  } 

  vpc_id      = "redacted"
  subnet_ids  = ["redacted"]

  endpoint_public_access  = true
  endpoint_private_access = true

  enable_cluster_creator_admin_permissions = true
  enable_irsa                              = true

  eks_managed_node_groups = {
    example = {
      ami_type            = "AL2023_ARM_64_STANDARD"
      instance_types      = ["t4g.medium"]
      min_size            = 2
      max_size            = 5
      desired_size        = 2
      labels = {
        "workload-type"                      = "management"
        "node-role.kubernetes.io/management" = "true"
      }

    }
  }

  tags = {
    "Environment" = "stage"
    "Project"     = "eks-test"
  }
}

[raficlosidus]

Nodes created Just fine
EKS Cluster created with no issues
All SG groups seem correct
The Node Group on the compute Tab on EKS is stuck in "CREATION" Mode and then fails .

[bryantbiggs]

You need a cluster DNS like coreDNS

[raficlosidus]

No Luck , Added this -

  addons = {

    coredns = {
      most_recent                 = true
      resolve_conflicts_on_create = "OVERWRITE"
      resolve_conflicts_on_update = "OVERWRITE"

    }
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

This is the whole TF file -

provider "aws" {
  region  = "us-east-1"
  profile = "stage"
}

module "eks_al2023" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "official-al2023"
  kubernetes_version = "1.34"

  # EKS Addons
  addons = {

    coredns = {
      most_recent                 = true
      resolve_conflicts_on_create = "OVERWRITE"
      resolve_conflicts_on_update = "OVERWRITE"

    }
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

  vpc_id     = "redacted"
  subnet_ids = ["redacted"]

  eks_managed_node_groups = {
    offical_nodes = {
      instance_types = ["m6i.large"]
      ami_type       = "AL2023_x86_64_STANDARD"
      min_size       = 2
      max_size       = 5
      desired_size   = 2

  }

}

[kgaweda]

I had the same issue as you — the problem is the vpc-cni plugin.

      addon_version  = "v1.20.3-eksbuild.1"
      most_recent    = false
      before_compute = true
    resolve_conflicts_on_create = "OVERWRITE"
    resolve_conflicts_on_update = "OVERWRITE"
    }

[raficlosidus]

@kgaweda
Still no luck I did what you suggested -

provider "aws" {
  region  = "us-east-1"
  profile = "stage"
}


# -------------------------------
# EKS Cluster Module
# -------------------------------
module "eks_al2023" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.8.0"

  name               = "stage-eks-cluster-new"
  kubernetes_version = "1.33"

  # EKS Addons
  addons = {
      coredns = {

      
      most_recent                 = true
      resolve_conflicts_on_create = "OVERWRITE"
      resolve_conflicts_on_update = "OVERWRITE"}
  
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {               # must apply before node creation
     addon_version  = "v1.20.3-eksbuild.1"
      most_recent                 = false
      resolve_conflicts_on_create = "OVERWRITE"
      resolve_conflicts_on_update = "OVERWRITE"
    }
  } 

  vpc_id      = "redacted"
  subnet_ids  = ["redacted"]

  endpoint_public_access  = true
  endpoint_private_access = true

  enable_cluster_creator_admin_permissions = true
  enable_irsa                              = true

  eks_managed_node_groups = {
    example = {
      ami_type            = "AL2023_ARM_64_STANDARD"
      instance_types      = ["t4g.medium"]
      min_size            = 2
      max_size            = 5
      desired_size        = 2
      labels = {
        "workload-type"                      = "management"
        "node-role.kubernetes.io/management" = "true"
      }

    }
  }

  tags = {
    "Environment" = "stage"
    "Project"     = "eks-test"
  }
}

bailed out after 30 minutes , Here is a response from Amazon Q -

What's happening: AWS is releasing new Kubernetes versions (1.33, 1.34) and new AMI types (AL2023) faster than they can properly test and validate all the combinations. The bootstrap tooling isn't keeping up with the release pace.

Why this happens:

• Kubernetes releases new versions every 3-4 months
• AWS tries to support them quickly to stay current
• AL2023 is relatively new and the tooling is still maturing
• ARM AMIs add another complexity layer
• Some combinations slip through testing

Your options right now:

1. Stick with proven combinations: AL2 + x86 + K8s 1.30
2. Report this to AWS Support - they need to know these AMIs are broken
3. Wait 2-4 weeks for AWS to fix the bootstrap tooling

The reliable stack that works:

version = "1.30"
--
ami_type = "AL2_x86_64"
instance_types = ["t3.medium"]

I know it's frustrating to have to downgrade, but AWS is essentially using customers as beta testers for these newer combinations. The older, proven stack will get your cluster working immediately.

[BryceBartmann]

I have the same issue - nodes never join and have this error

container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

[bryantbiggs]

I would check the VPC CNI addon in the console and see if it is reporting this error - I would also advise you open support tickets with AWS for this issue, its not a module related issue but something thats changed within EKS/VPC CNI

[bryantbiggs]

I had the same issue as you — the problem is the vpc-cni plugin.

      addon_version  = "v1.20.3-eksbuild.1"
      most_recent    = false
      before_compute = true
    resolve_conflicts_on_create = "OVERWRITE"
    resolve_conflicts_on_update = "OVERWRITE"
    }

This appears to fix the issue with the current plugin version - you don't have to pin the version but you will need

    resolve_conflicts_on_create = "OVERWRITE"
    resolve_conflicts_on_update = "OVERWRITE"

[champtar]

@bryantbiggs looking at #3218 my understanding is that you switched resolve_conflicts_on_create to NONE because it should not be needed, should we reconsider and put back the default to OVERWRITE ?

[bryantbiggs]

no - that doesn't make any sense in my opinion. if I enable the VPC CNI for the first time on a brand new cluster, what conflicts, if any, should exist?

More importantly, this CRD is an EKS provided/managed CRD

[champtar]

what conflicts, if any, should exist?

I 100% agree with you that there should be none, but here we are :)

[bryantbiggs]

if the API were created new today, I don't think the field resolve_conflicts_on_create would exist. I believe it is due to the legacy behavior of when EKS would deploy the required addons by default in an un-managed state and users wanted to then adopt those in their configurations. The action from the API perspective is named "create" but if the addon already existed in the cluster, albeit not managed by the API, then it was adopted and there could exist conflicts from modifications made that do not match the API defaults.

regardless, we aren't reversing the default behavior due to a single error on the AWS side; it also adds a decent amount of time to new cluster deploys https://github.com/terraform-aws-modules/terraform-aws-eks/tree/v20.37.2/tests/fast-addons