🌱 Implement nodeadm bootstrapping type #5700

faiq · 2025-10-07T20:30:21Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

This PR implements the nodeadm config type outlined by the KEP #5678

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

Release note:

Implement nodeadm bootstrapping type

k8s-ci-robot · 2025-10-07T20:30:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign neolit123 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

faiq · 2025-10-07T20:34:15Z

testing with this manifest

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: default-control-plane
  infrastructureRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: default-control-plane
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: default-control-plane
spec:
  addons:
  - name: kube-proxy
    version: v1.32.0-eksbuild.2
  network:
    cni:
      cniIngressRules:
      - description: kube-proxy metrics
        fromPort: 10249
        protocol: tcp
        toPort: 10249
      - description: NVIDIA Data Center GPU Manager metrics
        fromPort: 9400
        protocol: tcp
        toPort: 9400
      - description: Prometheus node exporter metrics
        fromPort: 9100
        protocol: tcp
        toPort: 9100
  region: us-west-2
  sshKeyName: ""
  version: v1.33.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: default
spec:
  template:
    spec:
      cloudInit:
        insecureSkipSecretsManager: true
      ami:
        eksLookupType: AmazonLinux2023
      instanceMetadataOptions:
        httpTokens: required
        httpPutResponseHopLimit: 2
      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
      instanceType: m5a.16xlarge
      rootVolume:
        size: 80
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: NodeadmConfigTemplate
metadata:
  name: default
spec:
  template:
    spec: {}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: default
spec:
  clusterName: default
  replicas: 1
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: NodeadmConfigTemplate
          name: default
      clusterName: default
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: default
      version: v1.33.0

faiq · 2025-10-07T20:36:59Z

/retest

faiq · 2025-10-08T16:59:02Z

/retest

faiq · 2025-10-08T19:13:01Z

/retest

faiq · 2025-10-08T20:53:23Z

/test ?

k8s-ci-robot · 2025-10-08T20:53:26Z

@faiq: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-aws-build

/test pull-cluster-api-provider-aws-build-docker

/test pull-cluster-api-provider-aws-e2e-blocking

/test pull-cluster-api-provider-aws-test

/test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-aws-apidiff-main

/test pull-cluster-api-provider-aws-e2e

/test pull-cluster-api-provider-aws-e2e-clusterclass

/test pull-cluster-api-provider-aws-e2e-conformance

/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts

/test pull-cluster-api-provider-aws-e2e-eks

/test pull-cluster-api-provider-aws-e2e-eks-gc

/test pull-cluster-api-provider-aws-e2e-eks-testing

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-aws-apidiff-main

pull-cluster-api-provider-aws-build

pull-cluster-api-provider-aws-build-docker

pull-cluster-api-provider-aws-e2e-blocking

pull-cluster-api-provider-aws-test

pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

faiq · 2025-10-08T20:53:56Z

/test /test pull-cluster-api-provider-aws-e2e-eks

dsanders1234 · 2025-10-09T18:29:11Z

Does this work with AWSManagedMachinePool ?

faiq · 2025-10-09T19:03:12Z

@dsanders1234 try this

 apiVersion: cluster.x-k8s.io/v1beta1
 kind: MachinePool
 metadata:
   name: default
 spec:
   clusterName: default
   template:
     spec:
       bootstrap:
         #dataSecretName: ""
         configRef:
           apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
           kind: NodeadmConfig
           name: default
       clusterName: default
       infrastructureRef:
         apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
         kind: AWSManagedMachinePool
         name: default
       version: v1.33.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedMachinePool
metadata:
  name: default
spec:
  roleName: "nodes.cluster-api-provider-aws.sigs.k8s.io"
  scaling:
    minSize: 1
    maxSize: 3
  amiType: CUSTOM
  awsLaunchTemplate:
    ami:
      eksLookupType: AmazonLinux2023
    instanceMetadataOptions:
      httpTokens: required
      httpPutResponseHopLimit: 2
    instanceType: "m5a.16xlarge"
    rootVolume:
      size: 80
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: NodeadmConfig
metadata:
  name: default
spec:
  kubelet:
    config:
      evictionHard:
        memory.available: "2000Mi"

faiq · 2025-10-09T19:03:24Z

/test pull-cluster-api-provider-aws-e2e-eks

faiq · 2025-10-09T20:32:26Z

/test pull-cluster-api-provider-aws-e2e-eks

faiq · 2025-10-09T20:40:44Z

/retest

faiq · 2025-10-09T20:52:21Z

/retest

faiq · 2025-10-09T21:01:01Z

/retest

faiq · 2025-10-09T21:16:52Z

/retest

faiq · 2025-11-11T18:34:48Z

/test pull-cluster-api-provider-aws-e2e-eks

matzegebbe · 2025-11-17T17:41:17Z

First of all, thanks for the work on this PR - I'm currently testing the build artifacts from this branch of @faiq in our environment.

During evaluation, I encountered RBAC-related errors such as:

nodeadmconfigs.bootstrap.cluster.x-k8s.io is forbidden: User "system:serviceaccount:capa-system:capa-controller-manager" cannot list resource "nodeadmconfigs" in API group "bootstrap.cluster.x-k8s.io" at the cluster scope

but we can of course fix that, along with a few other small stumbling blocks with the new CRDs and configuration.

While I'm working through these issues and completing our internal validation, I wanted to kindly ask whether there are any remaining blockers preventing this PR from being approved and merged.

EKS nodes images based on Amazon Linux 2 will go out of support on November 26 2025, and we are already running into additional problems with AL2 and the latest runc version (see: awslabs/amazon-eks-ami#2498
). In my opinion, ensuring that Cluster API and CAPA offer solid support for running fully supported machine images on AWS is critical for the project’s overall reliability and long-term relevance.

Thanks in advance for any update or clarification on the merge timeline ✌️

faiq · 2025-11-17T18:11:39Z

While I'm working through these issues and completing our internal validation, I wanted to kindly ask whether there are any remaining blockers preventing this PR from being approved and merged.

As far as I can tell, there are none other than the open source maintainers being busy both with their day jobs. If you can please push in the k8s slack for some reviews. Also, feel free to review this as well. All comments welcome.

faiq · 2025-11-26T17:08:58Z

/test pull-cluster-api-provider-aws-e2e-eks

faiq · 2025-11-26T17:14:05Z

/retest

faiq · 2025-11-26T17:50:43Z

/test pull-cluster-api-provider-aws-e2e-eks

faiq · 2025-11-26T18:01:11Z

/retest

faiq · 2025-11-26T19:24:54Z

/retest

faiq · 2025-11-26T23:33:22Z

/test pull-cluster-api-provider-aws-e2e-eks

AndiDog

I only reviewed a few pieces for now, might come back to this again later. Someone with more EKS experience would be great as reviewer.

AndiDog · 2025-11-28T09:25:23Z

templates/cluster-template-eks-nodeadm-clusterclass.yaml

@@ -0,0 +1,70 @@
+apiVersion: cluster.x-k8s.io/v1beta1


Can we please add a non-ClusterClass template as well?

I can. I feel like it will also need another set of tests, which seems like a lot here considering I've already added 2 e2e tests. Maybe we can follow up in another PR?

docs/book/src/topics/eks/creating-a-cluster.md

bootstrap/eks/internal/userdata/utils.go

faiq · 2025-12-01T16:18:04Z

/test pull-cluster-api-provider-aws-e2e-eks

faiq · 2025-12-01T16:50:00Z

/retest

faiq · 2025-12-01T17:13:57Z

/retest

faiq · 2025-12-01T19:44:52Z

/test pull-cluster-api-provider-aws-e2e-eks

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 7, 2025

k8s-ci-robot added the needs-priority label Oct 7, 2025

k8s-ci-robot requested review from AndiDog and nrb October 7, 2025 20:30

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 7, 2025

faiq force-pushed the faiq/nodeadm-upstream branch 2 times, most recently from 8f854bd to 81f3664 Compare October 8, 2025 20:43

faiq changed the title ~~🌱 feat: starts bootstrapping new type with kubebuilder~~ 🌱 feat: implements nodeadm bootstrapping type Oct 9, 2025

faiq force-pushed the faiq/nodeadm-upstream branch from 81f3664 to 59ecae0 Compare October 9, 2025 19:01

faiq force-pushed the faiq/nodeadm-upstream branch 2 times, most recently from 10503ae to 476eb43 Compare October 9, 2025 20:20

faiq force-pushed the faiq/nodeadm-upstream branch from 476eb43 to 234d905 Compare October 10, 2025 02:57

faiq force-pushed the faiq/nodeadm-upstream branch from 9f4ef81 to 483bbb3 Compare November 20, 2025 21:26

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 21, 2025

faiq force-pushed the faiq/nodeadm-upstream branch from 483bbb3 to 439664b Compare November 26, 2025 17:08

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 26, 2025

faiq force-pushed the faiq/nodeadm-upstream branch from 439664b to b685eea Compare November 26, 2025 17:42

faiq added 6 commits November 26, 2025 11:53

feat: adds nodeadm type

f25ffc2

e2e: increase eks parallelsim to 2 for e2e tests

5c1e318

e2e: adds flavor and e2e test for nodeadm clusterclass

63ab8fd

docs: adds documentation about nodeadm

0bdb879

fix: removes omitempty from nodeadm template

ef0cc1a

feat: rebase to v1.11 cluster api core

ebad622

faiq force-pushed the faiq/nodeadm-upstream branch from b685eea to 0dd3488 Compare November 26, 2025 23:32

AndiDog changed the title ~~🌱 feat: implements nodeadm bootstrapping type~~ 🌱 Implement nodeadm bootstrapping type Nov 28, 2025

AndiDog reviewed Nov 28, 2025

View reviewed changes

fixup! feat: rebase to v1.11 cluster api core

1ca7f1c

faiq force-pushed the faiq/nodeadm-upstream branch from 0dd3488 to 1ca7f1c Compare December 1, 2025 16:11

fix: addresses review commnets

2d348f8

🌱 Implement nodeadm bootstrapping type #5700

Are you sure you want to change the base?

🌱 Implement nodeadm bootstrapping type #5700

Conversation

faiq commented Oct 7, 2025 • edited by AndiDog Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 7, 2025

Uh oh!

faiq commented Oct 7, 2025

Uh oh!

faiq commented Oct 7, 2025

Uh oh!

faiq commented Oct 8, 2025

Uh oh!

faiq commented Oct 8, 2025

Uh oh!

faiq commented Oct 8, 2025

Uh oh!

k8s-ci-robot commented Oct 8, 2025

Uh oh!

faiq commented Oct 8, 2025

Uh oh!

dsanders1234 commented Oct 9, 2025

Uh oh!

faiq commented Oct 9, 2025

Uh oh!

faiq commented Oct 9, 2025

Uh oh!

faiq commented Oct 9, 2025

Uh oh!

faiq commented Oct 9, 2025

Uh oh!

faiq commented Oct 9, 2025

Uh oh!

faiq commented Oct 9, 2025

Uh oh!

faiq commented Oct 9, 2025

Uh oh!

faiq commented Nov 11, 2025

Uh oh!

matzegebbe commented Nov 17, 2025

Uh oh!

faiq commented Nov 17, 2025

Uh oh!

faiq commented Nov 26, 2025

Uh oh!

faiq commented Nov 26, 2025

Uh oh!

faiq commented Nov 26, 2025

Uh oh!

faiq commented Nov 26, 2025

Uh oh!

faiq commented Nov 26, 2025

Uh oh!

faiq commented Nov 26, 2025

Uh oh!

AndiDog left a comment

Choose a reason for hiding this comment

Uh oh!

AndiDog Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

faiq Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

faiq commented Dec 1, 2025

Uh oh!

faiq commented Dec 1, 2025

Uh oh!

faiq commented Dec 1, 2025

Uh oh!

faiq commented Dec 1, 2025

Uh oh!

Reviewers

Assignees

faiq commented Oct 7, 2025 •

edited by AndiDog

Loading