aws_eks_addon pod_identity_association

[cdenneen]

Errors:

updating EKS Add-On (dev-use1-400:cert-manager): operation error EKS: UpdateAddon, https response error StatusCode: 400, RequestID: REDACTED, InvalidParameterException: Pod Identity feature is not supported for addon version: v1.18.2-eksbuild.2

updating EKS Add-On (dev-use1-400:amazon-cloudwatch-observability): operation error EKS: UpdateAddon, https response error StatusCode: 400, RequestID: REDACTED, InvalidParameterException: Service account amazon-cloudwatch in pod identity configuration is not supported for addon amazon-cloudwatch-observability

I know the cert-manager one was working so not sure why that would be now, unless its a bug in the latest EKS add-on version of cert-manager?

The Service Account for amazon-cloudwatch is same one used in the pod-identity module but maybe the "hyphen" here is causing the issue with updating, does it need to be escaped or something?

Code:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.3.1"
...
  addons = {
    aws-efs-csi-driver = {
      most_recent                 = true
      resolve_conflicts_on_update = "OVERWRITE"
      pod_identity_association = [{
        role_arn        = try(module.aws_efs_csi_pod_identity.iam_role_arn, null)
        service_account = "efs-csi-controller-sa"
      }]
    }
    external-dns = {
      most_recent                 = true
      resolve_conflicts_on_update = "OVERWRITE"
      pod_identity_association = [{
        role_arn        = try(module.external_dns_pod_identity.iam_role_arn, null)
        service_account = "external-dns"
      }]
    }
    cert-manager = {
      most_recent                 = true
      resolve_conflicts_on_update = "OVERWRITE"
      pod_identity_association = [{
        role_arn        = try(module.cert_manager_pod_identity.iam_role_arn, null)
        service_account = "cert-manager"
      }]
    }
    amazon-cloudwatch-observability = {
      most_recent                 = true
      resolve_conflicts_on_update = "OVERWRITE"
      pod_identity_association = [{
        role_arn        = try(module.cloudwatch_pod_identity.iam_role_arn, null)
        service_account = "amazon-cloudwatch"
      }]
    }
    snapshot-controller = {
      most_recent                 = true
      resolve_conflicts_on_update = "OVERWRITE"
    }
  }
...
}

Plan

  ~ resource "aws_eks_addon" "this" {
        id                          = "dev-use1-400:amazon-cloudwatch-observability"
        tags                        = {
            "Environment"   = "production"
            "Service"       = "EKS"
            "Support Team"  = "Platform Engineering"
            "Terraform"     = "true"
        }
        # (11 unchanged attributes hidden)
      + pod_identity_association {
          + role_arn        = "arn:aws:iam::1234567890:role/dev-use1-400-AmazonEKS_Observability_Role"
          + service_account = "amazon-cloudwatch"
        }
        # (1 unchanged block hidden)
    }
  ~ resource "aws_eks_addon" "this" {
        id                          = "dev-use1-400:cert-manager"
        tags                        = {
            "Environment"   = "production"
            "Service"       = "EKS"
            "Support Team"  = "Platform Engineering"
            "Terraform"     = "true"
        }
        # (11 unchanged attributes hidden)
      + pod_identity_association {
          + role_arn        = "arn:aws:iam::1234567890:role/dev-use1-400-cert-manager"
          + service_account = "cert-manager"
        }
        # (1 unchanged block hidden)
    }

[bryantbiggs]

you will need to open a ticket with AWS for those issues - they are unrelated to the module. The module doesn't do anything other than pass values directly through to the API

[cdenneen]

@bryantbiggs so the eks-pod-identity module seems to create the association without issue whereas the eks one here setting up the associations like above give that error about cert-manager. When I checked the associations are being created from the eks-pod-identity module so not sure why that one works but coming from this module it doesn't... are they using 2 different APIs?

module "cert_manager_pod_identity" {
  source  = "terraform-aws-modules/eks-pod-identity/aws"
  version = "2.0.0"

  name               = "${module.eks.cluster_name}-cert-manager"
  use_name_prefix    = false
  policy_name_prefix = "${module.eks.cluster_name}-AmazonEKS_"
  attach_cert_manager_policy    = true
  association_defaults = {
    namespace       = "cert-manager"
    service_account = "cert-manager"
  }
  associations = {
    cluster = {
      cluster_name = module.eks.cluster_name
    }
  }
}

Only difference I see when using the eks-pod-identity module is it doesn't associate it with an Addon in the EKS Console where as when you do it in the addons hash it seems to create that hyperlink to the add-on.

Can you tell me is there a preference for adding the associations as part of the addons hash instead of the pod-identity module?
Does it make sense to remove the pod_identity_association = [{...}] blocks from the var.addons and just leverage the other module? Not sure what the plan is moving forward to keep both, deprecate, preferred way, etc.

Thanks

[bryantbiggs]

there are use cases for both - so both will continue to exist. however, any configuration settings on the EKS Addon API must be supported by the addon itself (i.e. - ADOT tends to not support new EKS versions for quite some time, unfortunately. Its also up to the addon authors to support the native pod identity association)