How to properly use ENABLE_PREFIX_DELEGATION in aws-node + eks + cdk

[storyalex]

Hello CDK people!

I'm trying to figure out how to have CDK create an EKS cluster that allows more max-pods using the prefix delegation as described here: https://aws.amazon.com/blogs/containers/amazon-vpc-cni-increases-pods-per-node-limits/

So far I can get it to work, but I have to deploy two times and my solution feels hacky at best.

My current setup steps:

1. Create EKS cluster with defaultCapacity: 0 to allow using a launch templates
2. Patch the aws-node daemonset to set environment ENABLE_PREFIX_DELEGATION=true
3. Set up a launch template that manually bootstraps EKS with custom parameters:

    const lt = new ec2.CfnLaunchTemplate(this, 'LaunchTemplate', {
      launchTemplateData: {
        userData: Fn.base64(`MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash

/etc/eks/bootstrap.sh ${cluster.clusterName} --use-max-pods false --kubelet-extra-args '--max-pods=110'

--==MYBOUNDARY==--\\
        `),
      },
    });

4. Create a defaultNodeGroup for the cluster that uses the launch template

After deploying I check max pods using kubectl describe nodes | grep pods: and get 17 (which I guess is default with t3.small).
Bootstrap command stolen from this page: https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html

Here comes the weird part:
If I do any kind of change to the launch template - adding a comment is enough - and deploy again I can see it adding another node to the cluster and removing the old one and now checking max pods the same way it gives me 110.

I didn't want to add this as a bug report or documentation issue since I probably did something wrong here.
Do you guys have any suggestions on how to do this the proper way? I can't find a proper way to translate the info on the pages I've linked to CDK code.

Open to all suggestions here!

// Alexander

[storyalex]

I created a test repository with a minimal example that shows what I experience.

https://github.com/storyalex/eks-test

Steps to reproduce:

1. Deploy
2. Change something in the launch template
3. Deploy again

After the first deployment capacity and allocatable pods are set to 17. After the second deployment, it is at 110 pods.

[storyalex]

I managed to solve this.

The problem was that I tried to set parameters in the cluster.defaultNodegroup which is not the same as cluster.addNodegroup (or w/e). And nodegroups do not take bootstrapOptions.

I switched to using an AutoscalingGroup, which takes bootstrapOptions (but can't use a launchtemplate - that's a problem for later though).

I realise using typescript instead of javascript might have helped me figure this out sooner. Let that be a lesson for you if you're just starting out with CDK!

Here's the change in the repo I posted: https://github.com/storyalex/eks-test/commit/13536568c7e693d8f93c8fc169b24d0ae4a25f73
After deploying that and checking with kubectl describe nodes | grep pods: I get 110.

[zentavr]

@storyalex your repo is not available anymore.