Disable EKS auto mode fails

[project0]

Description

Disabling eks auto mode on an existing cluster fails. Apparently there are some related changes and issues that claimed to solve and reverted it again. As all related issues are closed i am have decided to raise a new issues.

• disabling auto mode #3268
• fix: Correct Auto Mode disable #3253
• fix: Ability to desactivate Auto Mod #3252
• Terraform apply fails after importing EKS cluster with Auto Mode disabled #3236

Versions

• Module version [Required]:

• Terraform version:
  Terraform v1.10.2

• Provider version(s):

14:44:13.210 STDOUT terraform: Providers required by configuration:
14:44:13.210 STDOUT terraform: .
14:44:13.210 STDOUT terraform: ├── provider[registry.terraform.io/hashicorp/aws] >= 5.9.0
14:44:13.210 STDOUT terraform: ├── module.irsa_external-dns
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.0.0
14:44:13.210 STDOUT terraform: ├── module.karpenter
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: ├── module.vpc
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 5.46.0
14:44:13.210 STDOUT terraform: ├── module.eks
14:44:13.210 STDOUT terraform: │   ├── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │   ├── provider[registry.terraform.io/hashicorp/tls] >= 3.0.0
14:44:13.210 STDOUT terraform: │   ├── provider[registry.terraform.io/hashicorp/time] >= 0.9.0
14:44:13.210 STDOUT terraform: │   ├── module.eks_managed_node_group
14:44:13.210 STDOUT terraform: │       ├── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │       └── module.user_data
14:44:13.210 STDOUT terraform: │           ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
14:44:13.210 STDOUT terraform: │           └── provider[registry.terraform.io/hashicorp/null] >= 3.0.0
14:44:13.210 STDOUT terraform: │   ├── module.fargate_profile
14:44:13.210 STDOUT terraform: │       └── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │   ├── module.kms
14:44:13.210 STDOUT terraform: │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.33.0
14:44:13.210 STDOUT terraform: │   └── module.self_managed_node_group
14:44:13.210 STDOUT terraform: │       ├── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │       └── module.user_data
14:44:13.210 STDOUT terraform: │           ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
14:44:13.210 STDOUT terraform: │           └── provider[registry.terraform.io/hashicorp/null] >= 3.0.0
14:44:13.210 STDOUT terraform: ├── module.irsa_argocd
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.0.0
14:44:13.210 STDOUT terraform: └── module.irsa_aws-load-balancer-controller
14:44:13.210 STDOUT terraform:     └── provider[registry.terraform.io/hashicorp/aws] >= 4.0.0
14:44:13.210 STDOUT terraform: Providers required by state:
14:44:13.210 STDOUT terraform:     provider[registry.terraform.io/hashicorp/aws]
14:44:13.210 STDOUT terraform:     provider[registry.terraform.io/hashicorp/time]
14:44:13.210 STDOUT terraform:     provider[registry.terraform.io/hashicorp/tls]

Reproduction Code [Required]

working with auto mode

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31"

  cluster_name    = local.name
  cluster_version = "1.31"

  # auto mode
  cluster_compute_config = {
    enabled = true
    # see custom node pools in manifests/modules/cluster-aws-eks/
    node_pools = ["system"]
  }

  cluster_endpoint_public_access = true

  vpc_id            = module.vpc.vpc_id
  subnet_ids        = module.vpc.private_subnets
  cluster_ip_family = "ipv6"

  create_cni_ipv6_iam_policy = true
  iam_role_additional_policies = {
    "policy-eks-cluster" = aws_iam_policy.iam_cluster_policy.arn
  }
}

changing to:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31"

  cluster_name    = local.name
  cluster_version = "1.31"

  # auto mode
  #cluster_compute_config = {
  #  # disable auto mode
  #  enabled = false
  #  # see custom node pools in manifests/modules/cluster-aws-eks/
  ##  node_pools = ["system"]
  #}


  cluster_endpoint_public_access = true

  vpc_id            = module.vpc.vpc_id
  subnet_ids        = module.vpc.private_subnets
  cluster_ip_family = "ipv6"

  create_cni_ipv6_iam_policy = true
  iam_role_additional_policies = {
    "policy-eks-cluster" = aws_iam_policy.iam_cluster_policy.arn
  }


  eks_managed_node_group_defaults = {
    iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }

  eks_managed_node_groups = {
    system = {
      # https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html
      ami_type       = "BOTTLEROCKET_ARM_64"
      instance_types = ["t4g.large"]
      capacity_type  = "ON_DEMAND"
      min_size       = 1
      max_size       = 2
      desired_size   = 1

      labels = {
        # Used to ensure Karpenter runs on nodes that it does not manage
        "karpenter.sh/controller" = "true"
        "CriticalAddonsOnly"      = "true"
      }

      taints = {
        # The pods that do not tolerate this taint should run on nodes
        CriticalAddonsOnly = {
          key    = "CriticalAddonsOnly"
          value  = "true"
          effect = "NO_SCHEDULE"
        }
      }
    }
  }

}

Steps to reproduce the behavior:

Expected behavior

no error

Actual behavior

Error: compute_config.enabled, kubernetes_networking_config.elastic_load_balancing.enabled, and storage_config.block_storage.enabled must all be set to either true or false

Terminal Output Screenshot(s)

  │ Error: compute_config.enabled, kubernetes_networking_config.elastic_load_balancing.enabled, and storage_config.block_storage.enabled must all be set to either true or false
  │ 
  │   with module.eks.aws_eks_cluster.this[0],
  │   on .terraform/modules/eks/main.tf line 35, in resource "aws_eks_cluster" "this":
  │   35: resource "aws_eks_cluster" "this" {

Additional context

[colinjohnsonbeyond]

Just hit this issue myself

[mbmblbelt]

Same here

There are three structures in the cluster config that are created/modifed as a result of enabling auto mode.

• kubernetesNetworkConfig
• computeConfig
• storageConfig

Prior to enabling auto mode, kubernetesNetworkConfig does not have an elasticLoadBalancing setting and the computeConfig and storageConfig structures do not exist in the cluster config at all.

❯ aws eks describe-cluster --name <cluster name> | jq '.cluster'
{
  "name": <redacted>,
  "arn": <redacted>,
  "createdAt": "2022-11-30T09:31:35.889000-05:00",
  "version": "1.30",
  "endpoint": <redacted>,
  "roleArn": <redacted>,
  "resourcesVpcConfig": {
    "subnetIds": [
      <redacted>
    ],
    "securityGroupIds": [
      <redacted>
    ],
    "clusterSecurityGroupId": <redacted>,
    "vpcId": <redacted>,
    "endpointPublicAccess": true,
    "endpointPrivateAccess": true,
    "publicAccessCidrs": [
      <redacted>
    ]
  },
  "kubernetesNetworkConfig": {
    "serviceIpv4Cidr": <redacted>,
    "ipFamily": "ipv4"
  },
  "logging": {
    "clusterLogging": [
      {
        "types": [
          "api",
          "audit",
          "authenticator"
        ],
        "enabled": true
      },
      {
        "types": [
          "controllerManager",
          "scheduler"
        ],
        "enabled": false
      }
    ]
  },
  "identity": {
    "oidc": {
      "issuer": <redacted>
    }
  },
  "status": "ACTIVE",
  "certificateAuthority": {
    "data": <redacted>
  },
  "platformVersion": "eks.23",
  "tags": {
   <redacted>
  },
  "health": {
    "issues": []
  },
  "accessConfig": {
    "authenticationMode": "API_AND_CONFIG_MAP"
  },
  "upgradePolicy": {
    "supportType": "EXTENDED"
  }
}

After enabling auto mode, the elasticLoadBalancing setting is added to the kubernetesNetworkConfig structure and the computeConfig and storageConfig structures are created/added.

❯ aws eks describe-cluster --name <cluster name> | jq '.cluster'
{
  "name": <redacted>,
  "arn": <redacted>,
  "createdAt": "2022-11-30T09:31:35.889000-05:00",
  "version": "1.30",
  "endpoint": <redacted>,
  "roleArn": <redacted>,
  "resourcesVpcConfig": {
    "subnetIds": [
      <redacted>
    ],
    "securityGroupIds": [
      <redacted>
    ],
    "clusterSecurityGroupId": <redacted>,
    "vpcId": <redacted>,
    "endpointPublicAccess": true,
    "endpointPrivateAccess": true,
    "publicAccessCidrs": [
      <redacted>
    ]
  },
  "kubernetesNetworkConfig": {
    "serviceIpv4Cidr": <redacted>,
    "ipFamily": "ipv4",
    "elasticLoadBalancing": {
      "enabled": true
    }
  },
  "logging": {
    "clusterLogging": [
      {
        "types": [
          "api",
          "audit",
          "authenticator"
        ],
        "enabled": true
      },
      {
        "types": [
          "controllerManager",
          "scheduler"
        ],
        "enabled": false
      }
    ]
  },
  "identity": {
    "oidc": {
      "issuer": <redacted>
    }
  },
  "status": "ACTIVE",
  "certificateAuthority": {
    "data": <redacted>
  },
  "platformVersion": "eks.23",
  "tags": {
   <redacted>
  },
  "health": {
    "issues": []
  },
  "accessConfig": {
    "authenticationMode": "API_AND_CONFIG_MAP"
  },
  "upgradePolicy": {
    "supportType": "EXTENDED"
  },
  "computeConfig": {
    "enabled": true,
    "nodePools": ["system", "general-purpose"]
  },
  "storageConfig": {
    "blockStorage": {
      "enabled": true
    }
  }
}

These structures can not be removed once they have been added so simply commenting out the cluster_compute_config

# cluster_compute_config = {
#     enabled    = true
#     node_pools = ["system", "general-purpose"]
# }

or disabling it

cluster_compute_config = {
    enabled    = false
    node_pools = []
}

results in the module trying to set the storageConfig to null, which is not allowed now that the storageConfig structure exists in the cluster config

- storage_config {
          - block_storage {
              - enabled = false -> null
            }
        }

[reggora-mmatney]

@bryantbiggs I see that you were attempting to solve this problem in #3253 and had to revert in #3255

Are you still working on this problem? If so, do you have an estimate on when a fix will be released? Thanks

[bryantbiggs]

This has to be fixed in the provider first

[project0]

this issue looks pretty similar, but the error message is somehow different: hashicorp/terraform-provider-aws#40582

[bryantbiggs]

track hashicorp/terraform-provider-aws#41155

[lorengordon]

I've posted another attempt to address this problem, see #3308. I've confirmed it works to enable and disable Auto Mode, and enable it again. There are some limitations imposed by the API (not the provider), such that you can't change from a non-empty node_pools config to an empty node_pools config. But other than that, works fine.

[lorengordon]

Well I guess the PR doesn't update once closed, but the branch has the fix. You can review the changeset here, instead, master...lorengordon:terraform-aws-eks:fix/compute-config-false-or-null

[cmam-programize]

@bryantbiggs I get that one should track the issue on the provider, just a question though, how is this module compatible with what the latest published version of the provider documents? I mean that in the docs, it says:

When using EKS Auto Mode compute_config.enabled, kubernetes_network_config.elastic_load_balancing.enabled, and storage_config.block_storage.enabled must *ALL be set to true. Likewise for disabling EKS Auto Mode, all three arguments must be set to false. Enabling EKS Auto Mode also requires that bootstrap_self_managed_addons is set to false

However, in the module's code, this is not respected unless I'm missing something:

• https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/main.tf#L79
• https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/main.tf#L137

BTW, the use case I'm seeing this is when creating a new cluster.

Thanks!

[bryantbiggs]

What's not respected?

[cmam-programize]

The code generates the following blocks if auto_mode_enabled is false:

      + compute_config {
          + enabled = false
        }

      + kubernetes_network_config {
          + ip_family         = "ipv4"
          + service_ipv4_cidr = (known after apply)
          + service_ipv6_cidr = (known after apply)
        }
      # No storage_config block at all

I would expect the following, based on what I understand in the provider's doc though:

      compute_config {
          enabled = false
      }

      kubernetes_network_config {
           ...
           elastic_load_balancing {
               enabled = false
           }
      }

      storage_config {
          block_storage {
            enabled = false
          }
      }

Right?

(Edit: Added some missing brackets)