[EKS Controller] could not change the upgrade policy for cluster

[gecube]

I created cluster with the default properties like:

apiVersion: eks.services.k8s.aws/v1alpha1
kind: Cluster
metadata:
  name: production-new
  namespace: infra-production
spec:
  kubernetesNetworkConfig:
    ipFamily: ipv4
    serviceIPv4CIDR: 172.20.0.0/16
  name: production-new
  roleARN: arn:aws:iam::*****:role/eks-production-role
  logging:
    clusterLogging:
      - enabled: true
        types:
          - api
          - audit
          - authenticator
          - controllerManager
          - scheduler
  resourcesVPCConfig:
    endpointPrivateAccess: true
    endpointPublicAccess: false
    subnetIDs:
      - subnet-0eeac56411254cbc6
      - subnet-0f06902b47c880118
      - subnet-0c72af713be937dcc
  tags:
    Name: production-new

I found that it is created by default with Extended support policy:

I want to change the upgrade policy to standard so I am changing the manifest and apply. I am getting:

apiVersion: eks.services.k8s.aws/v1alpha1
kind: Cluster
metadata:
  resourceVersion: '936721627'
  name: production-new
  uid: db483972-9b13-4c93-a18a-252a2ec5f64c
  creationTimestamp: '2025-04-12T05:21:07Z'
  generation: 4
...
  namespace: infra-production
  finalizers:
    - finalizers.eks.services.k8s.aws/Cluster
  labels:
    kustomize.toolkit.fluxcd.io/name: infra-management
    kustomize.toolkit.fluxcd.io/namespace: flux-system
spec:
  resourcesVPCConfig:
    endpointPrivateAccess: true
    endpointPublicAccess: false
    subnetIDs:
      - subnet-0eeac56411254cbc6
      - subnet-0f06902b47c880118
      - subnet-0c72af713be937dcc
  accessConfig:
    authenticationMode: CONFIG_MAP
    bootstrapClusterCreatorAdminPermissions: true
  roleARN: 'arn:aws:iam::*****:role/eks-production-role'
  kubernetesNetworkConfig:
    elasticLoadBalancing:
      enabled: false
    ipFamily: ipv4
    serviceIPv4CIDR: 172.20.0.0/16
  name: production-new
  upgradePolicy:
    supportType: STANDARD
  version: '1.32'
  tags:
    Name: production-new
  logging:
    clusterLogging:
      - enabled: true
        types:
          - api
          - audit
          - authenticator
          - controllerManager
          - scheduler
status:
  platformVersion: eks.6
  ackResourceMetadata:
    arn: 'arn:aws:eks:eu-west-2:*****:cluster/production-new'
    ownerAccountID: '****'
    region: eu-west-2
  certificateAuthority:
    data: *****
  status: ACTIVE
  endpoint: 'https://******.eu-west-2.eks.amazonaws.com'
  conditions:
    - lastTransitionTime: '2025-04-29T08:13:37Z'
      status: 'True'
      type: ACK.ResourceSynced
    - message: 'InvalidParameterException: Cluster is already at the desired configuration with endpointPrivateAccess: true , endpointPublicAccess: false, and Public Endpoint Restrictions: [0.0.0.0/0]'
      status: 'True'
      type: ACK.Terminal
  createdAt: '2025-04-12T05:28:04Z'
  health: {}
  identity:
    oidc:
      issuer: 'https://oidc.eks.eu-west-2.amazonaws.com/id/*****'

so effectively upgrade policy is not changed

[github-actions]

Hello @gecube 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[gecube]

the logs of EKS controller are not very helpful

[gecube]

I was able to retrieve this after digging trough a billion lines:

{"level":"info","ts":"2025-04-29T08:13:37.922Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Cluster","namespace":"infra-production","name":"production-new","account":"******","role":"arn:aws:iam::*****:role/ack-ec2-controller-tooling","region":"eu-west-2","is_adopted":false,"generation":4,"diff":[{"Path":{"Parts":["Spec","AccessConfig","AuthenticationMode"]},"A":"CONFIG_MAP","B":"API_AND_CONFIG_MAP"},{"Path":{"Parts":["Spec","ResourcesVPCConfig","PublicAccessCIDRs"]},"A":null,"B":["0.0.0.0/0"]},{"Path":{"Parts":["Spec","UpgradePolicy","SupportType"]},"A":"STANDARD","B":"EXTENDED"}]}

[gecube]

after extending the original manifest, I was able to change the upgrade policy type, so effectively I needed to account all manual changes made to the cluster in the original manifest:

apiVersion: eks.services.k8s.aws/v1alpha1
kind: Cluster
metadata:
  name: production-new
  namespace: infra-production
spec:
  accessConfig:
    authenticationMode: API_AND_CONFIG_MAP
    bootstrapClusterCreatorAdminPermissions: true
  upgradePolicy:
    supportType: 'STANDARD'
  kubernetesNetworkConfig:
    ipFamily: ipv4
    serviceIPv4CIDR: 172.20.0.0/16
  name: production-new
  roleARN: arn:aws:iam::*****:role/eks-production-role
  logging:
    clusterLogging:
      - enabled: true
        types:
          - api
          - audit
          - authenticator
          - controllerManager
          - scheduler
  resourcesVPCConfig:
    endpointPrivateAccess: true
    endpointPublicAccess: false
    publicAccessCIDRs:
    - "0.0.0.0/0"
    subnetIDs:
      - subnet-0eeac56411254cbc6
      - subnet-0f06902b47c880118
      - subnet-0c72af713be937dcc
  tags:
    Name: production-new

[michaelhtm]

hello @gecube
I am unable to replicate this issue. When I create the cluster with the same configurations as yours
and later set the updateConfig, the update is successful. Is there something else that is happening here?

[gecube]

@michaelhtm Hi! As you can see there were manual changes in the cluster config effecting in drift between state in the cluster and the state in the cloud. The original cluster was created with default settings (so extended policy was applied). As the drift could not be resolved (IDK why) - the cluster upgrade policy is not applied. Anyway I still wonder why ACK does not overwrite manual changes on the cloud level. Maybe it is just because transition from API_AND_CONFIG_MAP to CONFIG_MAP is disallowed on API level (I know this because I saw a warning in cloud UI). So the main observation is that it is damn difficult to debug and fix. Also the status messages in the k8s resource are not very helpful.

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[gecube]

/remove-lifecycle stale