Skip to content

✨ IPv6 support for self-managed clusters #5603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tthvo wants to merge 26 commits into
base: main
Choose a base branch
from
Open

✨ IPv6 support for self-managed clusters #5603

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
6b4c826
validations: allow IPv6 configurations for unmanaged clusters
tthvo Jul 22, 2025
986b5e8
ec2: enable primary IPv6 on ENI for EC2 instances
tthvo Jul 22, 2025
4b48513
ec2: support option HTTPProtocolIPv6 for EC2 IMDS
tthvo Jul 22, 2025
e17adb5
routing: ensure routes to eigw are up to date
tthvo Jul 22, 2025
1c8b976
subnets: configure default subnets to use NAT64/DNS64
tthvo Jul 23, 2025
7d022ad
securitygroup: ensure icmpv6 is supported
tthvo Jul 23, 2025
e60c50c
securitygroup: allow setting allowed IPv6 CIDR for node NodePort serv…
tthvo Jul 28, 2025
d428141
securitygroup: allow configuring IPv6 source CIDRs for bastion SSH
tthvo Jul 28, 2025
6118462
crd: add IPv6 of bastion host to cluster status
tthvo Jul 30, 2025
3336db0
template: manifest templates for IPv6-enabled cluster
tthvo Jul 29, 2025
c795796
cni: customized calico manifests for single-stack IPv6
tthvo Jul 29, 2025
dff77ca
docs: add documentations for enabling IPv6 in non-eks clusters
tthvo Jul 29, 2025
65c25d5
validations: validate vpc and subnet CIDR
tthvo Aug 5, 2025
e21265d
docs: update doc for enabling ipv6
tthvo Aug 6, 2025
25bd540
cni: document the requirement for calico ipv6 support
tthvo Aug 8, 2025
50cac8e
subnets: wait till IPv6 CIDR is associated with subnets
tthvo Sep 19, 2025
73c25cf
sg: allow both ipv4 and ipv6 cidrs to API LB if vpc ipv6 block is def…
tthvo Sep 29, 2025
101c1c0
crd: clarify isIpv6 field on subnet spec
tthvo Jul 29, 2025
1cca7b9
api: add spec field to configure target group ipType
tthvo Oct 2, 2025
e0c6232
subnets: auto-assign IPv6 CIDR blocks to subnets when not specified
tthvo Oct 6, 2025
57d87ba
vpc: ipam pool under vpc.ipv6 should be used for VPC IPv6 CIDR
tthvo Oct 9, 2025
abe113a
subnets: only enable DNS64 for IPv6-only subnets
tthvo Oct 10, 2025
6f37668
docs: add dualstack cluster support documentation
tthvo Oct 10, 2025
e5dedfa
fixup! ec2: enable primary IPv6 on ENI for EC2 instances
tthvo Oct 23, 2025
1c0dc3d
fixup! docs: add dualstack cluster support documentation
tthvo Oct 23, 2025
160c6b1
fixup! api: add spec field to configure target group ipType
tthvo Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions pkg/cloud/services/interfaces.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ const (
AnyIPv4CidrBlock = "0.0.0.0/0"
// AnyIPv6CidrBlock is the CIDR block to match all IPv6 addresses.
AnyIPv6CidrBlock = "::/0"
// NAT64CidrBlock is the well-known CIDR block defined in RFC6052 for NAT64.
NAT64CidrBlock = "64:ff9b::/96"
)

// ASGInterface encapsulates the methods exposed to the machinepool
Expand Down
8 changes: 8 additions & 0 deletions pkg/cloud/services/network/routetables.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,13 @@ func (s *Service) getNatGatewayPrivateRoute(natGatewayID string) *ec2.CreateRout
}
}

func (s *Service) getNat64PrivateRoute(natGatewayID string) *ec2.CreateRouteInput {
return &ec2.CreateRouteInput{
NatGatewayId: aws.String(natGatewayID),
DestinationIpv6CidrBlock: aws.String(services.NAT64CidrBlock),
Copy link
Contributor

@mtulio mtulio Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is NAT64 required to make an operational cluster on AWS? Can we optionally enable it without impact, specially in preferred ipv6?

One of concerns of users willing to use IPv6 is the IPv4 costs, on AWS we can't fully eliminate it as public LBs still need it, but I wonder if we can reduce the dependency of Nat Gateways with that new proposal. What do you think?

With the preferred IPv6 topology, do we need to allocate one-per-az Nat Gateway (which is a expensive resource)? I wonder if we can create minimum topology (single, or dual, instead per-AZ) of NAT Gateways on IPv6 clusters, helping users to eliminate public IPv4 address from their environment.

topology

Something like: NATGatewayTopologyStrategy: default|single|dual

Where:

  • default: one per az, not changing the default (zonal redundant)
  • single: one NGW for entire cluster (SPOF)
  • dual: dual NGW with HA in two AZs (if cluster is deployed on >1 AZs)

}
}

func (s *Service) getEgressOnlyInternetGateway() *ec2.CreateRouteInput {
return &ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String(services.AnyIPv6CidrBlock),
Expand Down Expand Up @@ -415,6 +422,7 @@ func (s *Service) getRoutesToPrivateSubnet(sn *infrav1.SubnetSpec) (routes []*ec

routes = append(routes, s.getNatGatewayPrivateRoute(natGatewayID))
if sn.IsIPv6 {
routes = append(routes, s.getNat64PrivateRoute(natGatewayID))
Copy link

@sadasu sadasu Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Not sure why this was not needed for managed installs?
  2. Do we need to check for !sn.IsPublic here?

Copy link
Contributor Author

@tthvo tthvo Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check for !sn.IsPublic here?

I think the check is already done in

cluster-api-provider-aws/pkg/cloud/services/network/routetables.go

Lines 429 to 434 in 6f867be

func (s *Service) getRoutesForSubnet(sn *infrav1.SubnetSpec) ([]*ec2.CreateRouteInput, error) {
if sn.IsPublic {
return s.getRoutesToPublicSubnet(sn)
}
return s.getRoutesToPrivateSubnet(sn)
}

The func getRoutesToPrivateSubnet is only ever used for getting routes for private subnets :D

Not sure why this was not needed for managed installs

This route is actually a component of DNS64 (NAT64 is always on) to resolve an IPv4-only internet service to a synthetic IPv6 (See here). It is really a nice-to-have feature 🤔 but not required.

My guess was that when Ipv6 was supported for managed eks, they didn't need IPv6-only pods to access IPv4-only internet services.

Copy link
Contributor Author

@tthvo tthvo Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, I think CAPA currently support IPv6 for EKS by using dualstack subnets. And NAT64/DNS64 is really meant for IPv6-only subnets, not dualstack subnets (unless really necessary).

See details in commit: 78cb9d4

if !s.scope.VPC().IsIPv6Enabled() {
// Safety net because EgressOnlyInternetGateway needs the ID from the ipv6 block.
// if, for whatever reason by this point that is not available, we don't want to
Expand Down
18 changes: 18 additions & 0 deletions pkg/cloud/services/network/routetables_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,13 @@ func TestReconcileRouteTables(t *testing.T) {
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("64:ff9b::/96"),
NatGatewayId: aws.String("nat-01"),
RouteTableId: aws.String("rt-1"),
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("::/0"),
EgressOnlyInternetGatewayId: aws.String("eigw-01"),
Expand Down Expand Up @@ -247,6 +254,13 @@ func TestReconcileRouteTables(t *testing.T) {
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("64:ff9b::/96"),
NatGatewayId: aws.String("nat-01"),
RouteTableId: aws.String("rt-1"),
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("::/0"),
EgressOnlyInternetGatewayId: aws.String("eigw-01"),
Expand Down Expand Up @@ -1199,6 +1213,10 @@ func TestService_getRoutesForSubnet(t *testing.T) {
DestinationCidrBlock: aws.String("0.0.0.0/0"),
NatGatewayId: aws.String("nat-gw-fromZone-us-east-1a"),
},
{
DestinationIpv6CidrBlock: aws.String("64:ff9b::/96"),
NatGatewayId: aws.String("nat-gw-fromZone-us-east-1a"),
},
{
DestinationIpv6CidrBlock: aws.String("::/0"),
EgressOnlyInternetGatewayId: aws.String("vpc-eigw"),
Expand Down
22 changes: 22 additions & 0 deletions pkg/cloud/services/network/subnets.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,6 +545,28 @@ func (s *Service) createSubnet(sn *infrav1.SubnetSpec) (*infrav1.SubnetSpec, err
return nil, errors.Wrapf(err, "failed to set subnet %q attribute assign ipv6 address on creation", *out.Subnet.SubnetId)
}
record.Eventf(s.scope.InfraCluster(), "SuccessfulModifySubnetAttributes", "Modified managed Subnet %q attributes", *out.Subnet.SubnetId)

// Enable DNS64 so that the Route 53 Resolver returns DNS records for IPv4-only services
// containing a synthesized IPv6 address prefixed 64:ff9b::/96.
// This is needed alongside NAT64 to allow IPv6-only workloads to reach IPv4-only services.
// We only need to enable on private subnets.
if !sn.IsPublic {
Copy link
Contributor

@mtulio mtulio Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments on this:
1:

This is needed alongside NAT64 to allow IPv6-only workloads to reach IPv4-only services... We only need to enable on private subnets.

May this prevent ip6-only workloads running in nodes created in "public subnets" accessing resources in private subnets?

  1. I am not seeing this route in any of private subnets of clusters I've created both with primarily ipv4 and ipv6, I need to investigate closely in the output manifests or logs if I missed some error

if err := wait.WaitForWithRetryable(wait.NewBackoff(), func() (bool, error) {
if _, err := s.EC2Client.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
SubnetId: out.Subnet.SubnetId,
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
}); err != nil {
return false, err
}
return true, nil
}, awserrors.SubnetNotFound); err != nil {
record.Warnf(s.scope.InfraCluster(), "FailedModifySubnetAttributes", "Failed modifying managed Subnet %q attributes: %v", *out.Subnet.SubnetId, err)
return nil, errors.Wrapf(err, "failed to set subnet %q attribute enable dns64", *out.Subnet.SubnetId)
}
record.Eventf(s.scope.InfraCluster(), "SuccessfulModifySubnetAttributes", "Modified managed Subnet %q attributes", *out.Subnet.SubnetId)
}
}

// AWS Wavelength Zone's public subnets does not support to map Carrier IP address on launch, and
Expand Down
102 changes: 68 additions & 34 deletions pkg/cloud/services/network/subnets_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1781,14 +1781,6 @@ func TestReconcileSubnets(t *testing.T) {
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -1865,6 +1857,22 @@ func TestReconcileSubnets(t *testing.T) {
}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.DescribeAvailabilityZones(context.TODO(), gomock.Any()).
Return(&ec2.DescribeAvailabilityZonesOutput{
AvailabilityZones: []types.AvailabilityZone{
Expand Down Expand Up @@ -3656,15 +3664,6 @@ func TestReconcileSubnets(t *testing.T) {
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -3741,6 +3740,22 @@ func TestReconcileSubnets(t *testing.T) {
}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.DescribeAvailabilityZones(context.TODO(), gomock.Any()).
Return(&ec2.DescribeAvailabilityZonesOutput{
AvailabilityZones: []types.AvailabilityZone{
Expand Down Expand Up @@ -3903,15 +3918,6 @@ func TestReconcileSubnets(t *testing.T) {
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PublicSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PublicSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -3988,6 +3994,24 @@ func TestReconcileSubnets(t *testing.T) {
}, nil).
After(zone1PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PrivateSubnet)

// zone 2
m.DescribeAvailabilityZones(context.TODO(), &ec2.DescribeAvailabilityZonesInput{
ZoneNames: []string{"us-east-1c"},
Expand Down Expand Up @@ -4077,14 +4101,6 @@ func TestReconcileSubnets(t *testing.T) {
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PublicSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PublicSubnet)
m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -4160,6 +4176,24 @@ func TestReconcileSubnets(t *testing.T) {
},
}, nil).
After(zone2PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PrivateSubnet)
},
},
}
Expand Down