✨ IPv6 support for self-managed clusters #5603

tthvo · 2025-07-30T21:03:14Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

As of today, CAPA supports IPv6 on EKS, but not self-managed clusters. Thus, these changes bring IPv6 support for self-managed clusters, specifically single-stack IPv6 (no dualstack support yet)

Which issue(s) this PR fixes:

Fixes #2420 (part 2 for self-managed cluster, part 1 covers EKS)

Special notes for your reviewer:

API server: By default, API LB restrict access to only over IPv6 (overwrites allowed). You need a host with IPv6 to run CAPI/CAPA.
EC2 instance type: Only nitro-based instance type can support IPv6.
CNI: The CNI plugin needs to support IPV6. I include a sample manifest in test/e2e/data/cni/calico_ipv6.yaml. Calico does not support IPv6 with "IP-in-IP" so we need to use VXLAN.

Checklist:

Release note:

Enable IPv6 support for self-managed kubernetes clusters

linux-foundation-easycla · 2025-07-30T21:03:20Z

The committers listed above are authorized under a signed CLA.

✅ login: tthvo / name: Thuan Vo (db21fa8, 21990e2, dab54f9, 195d236, fb949b4, 9582e18, 36585e2, e493be2, 3e9cf47, 280a160, 68d1fbd, 2eff20e, a88f5e5, fb826fd, 507ea5c, a02faf2, 59cf5cc, 225703c)

k8s-ci-robot · 2025-07-30T21:03:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nrb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-07-30T21:03:23Z

Welcome @tthvo!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-07-30T21:03:24Z

Hi @tthvo. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tthvo · 2025-07-30T21:23:21Z

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

k8s-ci-robot · 2025-07-30T21:23:24Z

@tthvo: GitHub didn't allow me to request PR reviews from the following users: sadasu, patrickdillon.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tthvo · 2025-07-30T21:33:01Z

A quick preview of kubectl get for an IPv6 self-managed cluster can be found here. So far, nodes, pods and services are assigned the expected IPv6 family IPs 😄

damdo · 2025-08-04T09:45:03Z

/ok-to-test

damdo · 2025-08-04T09:46:01Z

/assign @mtulio

Asking you for a review Marco as I know you have been working on this downstream

api/v1beta2/defaults.go

docs/book/src/topics/ipv6-enabled-cluster.md

nrb · 2025-08-05T18:48:57Z

pkg/cloud/services/securitygroup/securitygroups.go

@@ -53,7 +53,7 @@ const (
 	IPProtocolICMP = "icmp"

 	// IPProtocolICMPv6 is how EC2 represents the ICMPv6 protocol in ingress rules.
-	IPProtocolICMPv6 = "58"
+	IPProtocolICMPv6 = "icmpv6"


Could this change break existing clusters that are already using the previous value?

Ah, I believe not. The accepted values 58 is still the same on the AWSCluster CRD.

This value here is only used internally and I have a block here to convert it from icmpv6 to 58 before proceeding further: https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5603/files#diff-59c59dbf183fdde59cf0030bd23388381951860f0002a073a55f330c4733f8ddR932-R937

pkg/cloud/services/securitygroup/securitygroups_test.go

api/v1beta2/awscluster_webhook.go

api/v1beta2/awscluster_webhook_test.go

api/v1beta2/defaults.go

api/v1beta2/types.go

mtulio · 2025-08-08T15:06:55Z

pkg/cloud/services/ec2/instances.go

+		// When registering targets by instance ID for an IPv6 target group,
+		// the targets must have an assigned primary IPv6 address.
+		if s.scope.VPC().IsIPv6Enabled() {
+			netInterface.Ipv6AddressCount = aws.Int32(1)
+			netInterface.PrimaryIpv6 = aws.Bool(true)


Is there a scenario when users wants non-primary IPv6? example using ipv6 only in internet-facing LBs, and private subnets still using only ipv4?

I did have a thought about it, but the target group (i.e. existing logic) is being created with IpType IPv6 when IPv6 is enabled; thus this primary IPv6 setting is required.

cluster-api-provider-aws/pkg/cloud/services/elb/loadbalancer.go

Lines 1783 to 1785 in 5f62ff6

if s.scope.VPC().IsIPv6Enabled() {

targetGroupInput.IpAddressType = elbv2types.TargetGroupIpAddressTypeEnumIpv6

}

I guess the subnets used for IPv6 (or dualstack) should be a dual-stack subnets?

mtulio · 2025-08-08T15:19:56Z

pkg/cloud/services/network/routetables.go

@@ -170,6 +176,36 @@ func (s *Service) fixMismatchedRouting(specRoute *ec2.CreateRouteInput, currentR
 	return nil
 }

+func (s *Service) fixMissingRoutes(specRoutes []*ec2.CreateRouteInput, currentRoutes []types.Route, rt types.RouteTable) error {


I am not sure if I followed the requirement of this function. Are this failing into target route conflicts in the routes added by CAPA on default? or is it a additional transformation preventing incorrect APIs failures?

The issue I observed is that when IPv6 is enabled, a route to egress-only-internet-gateway is missing even though it should be added according to the below code.

cluster-api-provider-aws/pkg/cloud/services/network/routetables.go

Lines 416 to 424 in 5f62ff6

if sn.IsIPv6 {

if !s.scope.VPC().IsIPv6Enabled() {

// Safety net because EgressOnlyInternetGateway needs the ID from the ipv6 block.

// if, for whatever reason by this point that is not available, we don't want to

// panic because of a nil pointer access. This should never occur. Famous last words though.

return routes, errors.Errorf("ipv6 block missing for ipv6 enabled subnet, can't create route for egress only internet gateway")

}

routes = append(routes, s.getEgressOnlyInternetGateway())

}

I also observed that sn.IsIPv6 originally returned false in the first reconciliation or so, but later returned true as expected. This lead to the missing route to egress-only-internet-gateway for ::/0 as CAPA didn't make sure desired routes are available.

The consequences is that cloudinit failed to finish in the EC2 nodes as AWS API creds are invalid due to invalid timestamp (i.e. instance failed to sync time with AWS NTP) 🤔

I also observed that sn.IsIPv6 originally returned false in the first reconciliation or so, but later returned true as expected.

IIUC, there is a small time gap, where the IPv6 is not yet associated with the subnet. Thus, CAPA does not add the expected route initially. So, this function ensures that eventually, that route is added.

(AWSCluster resource)

AWS requires that when registering targets by instance ID for an IPv6 target group, the targets must have an assigned primary IPv6 address. Note: The default subnets managed by CAPA are already set up to assign IPv6 addresses to newly created ENIs.

The httpProtocolIPv6 field enables or disables the IPv6 endpoint of the instance metadata service. The SDK only applies this field if httpEndpoint is enabled. When running on single-stack IPv6, pods only have IPv6, thus requiring an IPv6 endpoint to query IMDS as IPv4 network is unreachable.

There is a brief period where the IPv6 CIDR is not yet associated with the subnets. Thus, CAPA initially creates a route table without a route to eigw. This change ensures that later reconcilation will add that missing route. Note: Route to eigw for destination "::/0" to eigw is required for EC2 instance time sync on start-up.

This allows IPv6-only workloads to reach IPv4-only services. AWS supports this via NAT64/DNS64. More details: https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html

The API for DescribeEgressOnlyInternetGateways does not support attachment.vpc-id filter. Thus, the call will return all available eigw. Consequences: - CAPA incorrectly selects an unintended eigw for use. Leading to route creation failure since the eigw belongs to a different VPC. - CAPA incorrectly destroys all eigw of all VPCs. This is very catastrophic as it can break other workloads. This commit changes the filter to use cluster tag instead. Additional safeguard is also included to check if the eigw is truly attached the VPC.

CAPA handles icmpv6 as a protocol number 58. AWS accepts protocol number when creating rules. However, describing a rule from AWS API returns the protocol name, thus causing CAPA to not recognize it and fail.

…ices For IPv4, we have field NodePortIngressRuleCidrBlocks that specifies the allowed source IPv4 CIDR for node NodePort services on port 30000-32767. This extends that field to also accept IPv6 source CIDRs.

We need an option to configure IPv6 source CIDRs for SSH ingress rule of the bastion host. This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.

When creating a bastion host for an IPv6 cluster, the instance has both public IPv4 and IPv6. Thus, we need to report them in the cluster status if any. This also adds an additional print column to display that bastion IPv6.

This is a minimal template set to install an IPv6-enabled cluster. Both the controlplane and worker nodes must use nitro-based instance type (with IPv6 support).

This is a set of customized calico CNI manifests to support single-stack IPv6 cluster. Note that VXLAN is used since IP-in-IP currently only supports IPv4. References: - https://docs.tigera.io/calico/latest/networking/ipam/ipv6#ipv6 - https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/config-options#switching-from-ip-in-ip-to-vxlan - https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip

This combines existing docs for IPv6 EKS clusters with non-EKS ones, and also properly register the topic page into the documentation TOC.

Validation for specified VPC and subnet CIDRs is added for early feedback from the webhook. There are already existing checks for bastion and nodePort CIDRs.

The following is added: - [BYO VPC] Mention the required route when enabling DNS64. - [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual stack VPC.

tthvo · 2025-08-08T21:05:54Z

/test pull-cluster-api-provider-aws-e2e-blocking

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Jul 30, 2025

k8s-ci-robot requested review from AndiDog and damdo July 30, 2025 21:03

k8s-ci-robot added needs-priority cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 30, 2025

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025

k8s-ci-robot requested a review from nrb July 30, 2025 21:23

tthvo force-pushed the singestack-ipv6 branch from ddacd99 to 13e4379 Compare July 31, 2025 00:37

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 4, 2025

k8s-ci-robot assigned mtulio Aug 4, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 4, 2025

tthvo force-pushed the singestack-ipv6 branch from 13e4379 to 8816d87 Compare August 5, 2025 18:09

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2025

tthvo force-pushed the singestack-ipv6 branch from 8816d87 to 5dd2888 Compare August 5, 2025 18:11

nrb reviewed Aug 5, 2025

View reviewed changes

api/v1beta2/defaults.go Outdated Show resolved Hide resolved

nrb reviewed Aug 5, 2025

View reviewed changes

docs/book/src/topics/ipv6-enabled-cluster.md Outdated Show resolved Hide resolved

nrb reviewed Aug 5, 2025

View reviewed changes

pkg/cloud/services/securitygroup/securitygroups_test.go Outdated Show resolved Hide resolved

nrb reviewed Aug 5, 2025

View reviewed changes

api/v1beta2/awscluster_webhook.go Show resolved Hide resolved

tthvo force-pushed the singestack-ipv6 branch from 26079ce to 30fd6b1 Compare August 5, 2025 21:30

mtulio reviewed Aug 8, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 8, 2025

tthvo added 6 commits August 8, 2025 12:13

validations: allow IPv6 configurations for unmanaged clusters

3e9cf47

(AWSCluster resource)

subnets: configure default subnets to use NAT64/DNS64

dab54f9

This allows IPv6-only workloads to reach IPv4-only services. AWS supports this via NAT64/DNS64. More details: https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html

tthvo force-pushed the singestack-ipv6 branch from 77b637d to 6113954 Compare August 8, 2025 19:22

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 8, 2025

tthvo added 12 commits August 8, 2025 12:55

securitygroup: ensure icmpv6 is supported

fb949b4

CAPA handles icmpv6 as a protocol number 58. AWS accepts protocol number when creating rules. However, describing a rule from AWS API returns the protocol name, thus causing CAPA to not recognize it and fail.

securitygroup: allow setting allowed IPv6 CIDR for node NodePort serv…

fb826fd

…ices For IPv4, we have field NodePortIngressRuleCidrBlocks that specifies the allowed source IPv4 CIDR for node NodePort services on port 30000-32767. This extends that field to also accept IPv6 source CIDRs.

securitygroup: allow configuring IPv6 source CIDRs for bastion SSH

db21fa8

We need an option to configure IPv6 source CIDRs for SSH ingress rule of the bastion host. This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.

crd: add IPv6 of bastion host to cluster status

280a160

When creating a bastion host for an IPv6 cluster, the instance has both public IPv4 and IPv6. Thus, we need to report them in the cluster status if any. This also adds an additional print column to display that bastion IPv6.

template: manifest templates for IPv6-enabled cluster

2eff20e

This is a minimal template set to install an IPv6-enabled cluster. Both the controlplane and worker nodes must use nitro-based instance type (with IPv6 support).

docs: regenerate crd reference doc

507ea5c

docs: add documentations for enabling IPv6 in non-eks clusters

e493be2

This combines existing docs for IPv6 EKS clusters with non-EKS ones, and also properly register the topic page into the documentation TOC.

crd: regenerate CRD and docs

a02faf2

validations: validate vpc and subnet CIDR

59cf5cc

Validation for specified VPC and subnet CIDRs is added for early feedback from the webhook. There are already existing checks for bastion and nodePort CIDRs.

docs: update doc for enabling ipv6

a88f5e5

The following is added: - [BYO VPC] Mention the required route when enabling DNS64. - [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual stack VPC.

cni: document the requirement for calico ipv6 support

68d1fbd

tthvo force-pushed the singestack-ipv6 branch from 6113954 to 68d1fbd Compare August 8, 2025 20:18

	if s.scope.VPC().IsIPv6Enabled() {
	targetGroupInput.IpAddressType = elbv2types.TargetGroupIpAddressTypeEnumIpv6
	}

	if sn.IsIPv6 {
	if !s.scope.VPC().IsIPv6Enabled() {
	// Safety net because EgressOnlyInternetGateway needs the ID from the ipv6 block.
	// if, for whatever reason by this point that is not available, we don't want to
	// panic because of a nil pointer access. This should never occur. Famous last words though.
	return routes, errors.Errorf("ipv6 block missing for ipv6 enabled subnet, can't create route for egress only internet gateway")
	}
	routes = append(routes, s.getEgressOnlyInternetGateway())
	}

✨ IPv6 support for self-managed clusters #5603

Are you sure you want to change the base?

✨ IPv6 support for self-managed clusters #5603

Conversation

tthvo commented Jul 30, 2025

Uh oh!

linux-foundation-easycla bot commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tthvo commented Aug 8, 2025

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Jul 30, 2025 •

edited

Loading