AWSMachines not getting created for worker nodes created using AWSMachinePool

[Archisman-Mridha]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

I created a MachinePool with some user-defined labels under .spec.template.metadata.labels. The self-managed K8s cluster got provisioned in AWS. The AWSMachine resource corresponding to the control plane EC2 instance got created.

But, for the worker node EC2 instance (managed by that MachinePool), the AWSMachine resource isn't getting created.

CAPA controller manager pod logs :

I1104 09:11:48.845572       1 autoscalinggroup.go:133] "Looking for asg by name" controller="awsmachinepool" con │
│ trollerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachinePool" AWSMachinePool="capi-cluster/kube │
│ aid-demo-primary" namespace="capi-cluster" reconcileID="e491d561-f2f4-4ece-ba85-841e3db3f4b6" machinePool="capi- │
│ cluster/kubeaid-demo-primary" cluster="capi-cluster/kubeaid-demo" name="kubeaid-demo-primary"                    │
│ I1104 09:11:50.273043       1 launchtemplate.go:73] "checking for existing launch template" controller="awsmachi │
│ nepool" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachinePool" AWSMachinePool="capi-c │
│ luster/kubeaid-demo-primary" namespace="capi-cluster" name="kubeaid-demo-primary" reconcileID="e491d561-f2f4-4ec │
│ e-ba85-841e3db3f4b6" machinePool="capi-cluster/kubeaid-demo-primary" cluster="capi-cluster/kubeaid-demo"         │
│ I1104 09:11:54.181752       1 awsmachine_controller.go:695] "Deleting unneeded entry from AWS Secret" secretPref │
│ ix="aws.cluster.x-k8s.io/45cd5fb6-2939-44d3-825b-95c3f82c784e"   
"Looking for asg by name" controller="awsmachinepool" controllerGroup="infrastructure.cluster.x-k8s.io" controllerK │
│ I1104 07:35:41.009690       1 launchtemplate.go:73] "checking for existing launch template" controller="aws

The pod keeps logging Looking for asg by name and checking for existing launch template.

Interestingly, if I remove those user-defined labels from under machinePool.spec.template.metadata.labels, everything works perfectly. The AWSMachine resource corresponding to the worker node EC2 instance gets created. And the CAPA controller manager stops logging after Deleting unneeded entry from AWS Secret.

What did you expect to happen:

After the cluster got provisioned, the AWSMachine resource corresponding to the worker node EC2 instance needs to get created.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster API Provider AWS (CAPA) version: v2.7.1
• Cluster API (CAPI) version: v1.7.2
• Kubernetes version: v1.31.0
• OS: Ubuntu 24.04

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Archisman-Mridha]

Update

This happens even when I don't specify any labels under the .spec.template.metadata.labels field in the MachinePool.

The worker nodes do get provisioned. But then the AWSMachine resources for those worker nodes never gets created in the management cluster.

This prevents creation of the Machine resources for those worker nodes as well.

Here are the manifests :

MachinePool :

apiVersion: v1
items:
- apiVersion: cluster.x-k8s.io/v1beta1
  kind: MachinePool
  metadata:
    annotations:
      cluster.x-k8s.io/replicas-managed-by: external-autoscaler
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"cluster.x-k8s.io/v1beta1","kind":"MachinePool","metadata":{"annotations":{"cluster.x-k8s.io/replicas-managed-by":"external-autoscaler"},"labels":{"argocd.argoproj.io/instance":"capi-cluster"},"name":"kubeaid-demo-primary","namespace":"capi-cluster"},"spec":{"clusterName":"kubeaid-demo","replicas":1,"template":{"metadata":{"labels":{"node-role.kubernetes.io/bootstrapper":"","node.cluster.x-k8s.io/machinepool":"primary"}},"spec":{"bootstrap":{"configRef":{"apiVersion":"bootstrap.cluster.x-k8s.io/v1beta1","kind":"KubeadmConfig","name":"kubeaid-demo-primary"}},"clusterName":"kubeaid-demo","infrastructureRef":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","kind":"AWSMachinePool","name":"kubeaid-demo-primary"},"version":"v1.31.0"}}}}
    creationTimestamp: "2024-11-05T12:10:58Z"
    finalizers:
    - machinepool.cluster.x-k8s.io
    generation: 3
    labels:
      argocd.argoproj.io/instance: capi-cluster
      cluster.x-k8s.io/cluster-name: kubeaid-demo
    name: kubeaid-demo-primary
    namespace: capi-cluster
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: kubeaid-demo
      uid: ab31c332-4c64-4f9b-b254-1c0d00855183
    resourceVersion: "4387"
    uid: 222db850-12e6-4b8a-b569-f390e08257f0
  spec:
    clusterName: kubeaid-demo
    minReadySeconds: 0
    providerIDList:
    - aws:///us-east-1c/i-08d5e93d21bc2e421
    replicas: 1
    template:
      metadata:
        labels:
          node-role.kubernetes.io/bootstrapper: ""
          node.cluster.x-k8s.io/machinepool: primary
      spec:
        bootstrap:
          configRef:
            apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
            kind: KubeadmConfig
            name: kubeaid-demo-primary
            namespace: capi-cluster
          dataSecretName: kubeaid-demo-primary
        clusterName: kubeaid-demo
        infrastructureRef:
          apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
          kind: AWSMachinePool
          name: kubeaid-demo-primary
          namespace: capi-cluster
        version: v1.31.0
  status:
    availableReplicas: 1
    bootstrapReady: true
    conditions:
    - lastTransitionTime: "2024-11-05T12:18:49Z"
      status: "True"
      type: Ready
    - lastTransitionTime: "2024-11-05T12:18:35Z"
      status: "True"
      type: BootstrapReady
    - lastTransitionTime: "2024-11-05T12:18:49Z"
      status: "True"
      type: InfrastructureReady
    - lastTransitionTime: "2024-11-05T12:10:59Z"
      status: "True"
      type: ReplicasReady
    infrastructureReady: true
    nodeRefs:
    - apiVersion: v1
      kind: Node
      name: ip-10-14-3-158.ec2.internal
      uid: dcb6207d-d346-4780-bf94-e8c2b2c0d47d
    observedGeneration: 3
    phase: Running
    readyReplicas: 1
    replicas: 1
kind: List
metadata:
  resourceVersion: ""

AWSMachinePool :

apiVersion: v1
items:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
  kind: AWSMachinePool
  metadata:
    annotations:
      cluster-api-provider-aws: "true"
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","kind":"AWSMachinePool","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"capi-cluster"},"name":"kubeaid-demo-primary","namespace":"capi-cluster"},"spec":{"awsLaunchTemplate":{"ami":{"id":"ami-0fc044fa0d061d18d"},"iamInstanceProfile":"nodes.cluster-api-provider-aws.sigs.k8s.io","imageLookupBaseOS":"ubuntu-20.04","instanceType":"t4g.medium","rootVolume":{"size":35},"sshKeyName":"kubeaid-demo"},"maxSize":3,"minSize":1}}
    creationTimestamp: "2024-11-05T12:10:59Z"
    finalizers:
    - awsmachinepool.infrastructure.cluster.x-k8s.io
    generation: 4
    labels:
      argocd.argoproj.io/instance: capi-cluster
      cluster.x-k8s.io/cluster-name: kubeaid-demo
    name: kubeaid-demo-primary
    namespace: capi-cluster
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: MachinePool
      name: kubeaid-demo-primary
      uid: 222db850-12e6-4b8a-b569-f390e08257f0
    resourceVersion: "5937"
    uid: 8b39c1b3-fcbc-4906-805d-184fc97e0db0
  spec:
    awsLaunchTemplate:
      ami:
        id: ami-0fc044fa0d061d18d
      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
      imageLookupBaseOS: ubuntu-20.04
      instanceType: t4g.medium
      rootVolume:
        deviceName: /dev/sda1
        size: 35
      sshKeyName: kubeaid-demo
    defaultCoolDown: 5m0s
    defaultInstanceWarmup: 5m0s
    maxSize: 3
    minSize: 1
    providerID: arn:aws:autoscaling:us-east-1:762233771014:autoScalingGroup:c5d57210-3c0b-4336-a070-9515575eeac5:autoScalingGroupName/kubeaid-demo-primary
    providerIDList:
    - aws:///us-east-1c/i-08d5e93d21bc2e421
  status:
    conditions:
    - lastTransitionTime: "2024-11-05T12:18:49Z"
      status: "True"
      type: Ready
    - lastTransitionTime: "2024-11-05T12:18:49Z"
      status: "True"
      type: ASGReady
    - lastTransitionTime: "2024-11-05T12:18:39Z"
      status: "True"
      type: LaunchTemplateReady
    instances:
    - instanceID: i-08d5e93d21bc2e421
    launchTemplateID: lt-0ef56ce4c49ee71f2
    launchTemplateVersion: "2"
    ready: true
    replicas: 1
kind: List
metadata:
  resourceVersion: ""

[fiunchinho]

Currently, the CAPA MachinePool implementation does not create AWSMachine resources. It creates AWS Auto Scaling Groups, and it lets the ASG manage the individual machines. So you won't find any Machines / AWSMachines for worker nodes, and it's working as expected.

[AndiDog]

See #5174

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten