Skip to content

✨ IPv6 support for self-managed clusters #5603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
+33,483 −5,671
Open

✨ IPv6 support for self-managed clusters #5603

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
7949ccb
validations: allow IPv6 configurations for unmanaged clusters
tthvo Jul 22, 2025
c1bafb0
ec2: enable primary IPv6 on ENI for EC2 instances
tthvo Jul 22, 2025
842f85a
ec2: support option HTTPProtocolIPv6 for EC2 IMDS
tthvo Jul 22, 2025
60308b2
routing: ensure routes to eigw are up to date
tthvo Jul 22, 2025
7826e7e
subnets: configure default subnets to use NAT64/DNS64
tthvo Jul 23, 2025
987bc9c
securitygroup: ensure icmpv6 is supported
tthvo Jul 23, 2025
7bbae60
securitygroup: allow setting allowed IPv6 CIDR for node NodePort serv…
tthvo Jul 28, 2025
cd6c178
securitygroup: allow configuring IPv6 source CIDRs for bastion SSH
tthvo Jul 28, 2025
1ebffda
crd: add IPv6 of bastion host to cluster status
tthvo Jul 30, 2025
2c7e1e4
template: manifest templates for IPv6-enabled cluster
tthvo Jul 29, 2025
da96854
cni: customized calico manifests for single-stack IPv6
tthvo Jul 29, 2025
cd7c9b2
docs: add documentations for enabling IPv6 in non-eks clusters
tthvo Jul 29, 2025
def75f2
validations: validate vpc and subnet CIDR
tthvo Aug 5, 2025
9851222
docs: update doc for enabling ipv6
tthvo Aug 6, 2025
ea41d67
cni: document the requirement for calico ipv6 support
tthvo Aug 8, 2025
bfbbb84
subnets: wait till IPv6 CIDR is associated with subnets
tthvo Sep 19, 2025
4d38ed3
sg: allow both ipv4 and ipv6 cidrs to API LB if vpc ipv6 block is def…
tthvo Sep 29, 2025
e5e3ad3
crd: clarify isIpv6 field on subnet spec
tthvo Jul 29, 2025
04d4286
api: add spec field to configure target group ipType
tthvo Oct 2, 2025
b8ca23a
subnets: auto-assign IPv6 CIDR blocks to subnets when not specified
tthvo Oct 6, 2025
3292f75
vpc: ipam pool under vpc.ipv6 should be used for VPC IPv6 CIDR
tthvo Oct 9, 2025
78cb9d4
subnets: only enable DNS64 for IPv6-only subnets
tthvo Oct 10, 2025
061134a
docs: add dualstack cluster support documentation
tthvo Oct 10, 2025
4701b61
fixup! ec2: enable primary IPv6 on ENI for EC2 instances
tthvo Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions pkg/cloud/services/interfaces.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ const (
AnyIPv4CidrBlock = "0.0.0.0/0"
// AnyIPv6CidrBlock is the CIDR block to match all IPv6 addresses.
AnyIPv6CidrBlock = "::/0"
// NAT64CidrBlock is the well-known CIDR block defined in RFC6052 for NAT64.
NAT64CidrBlock = "64:ff9b::/96"
)

// ASGInterface encapsulates the methods exposed to the machinepool
Expand Down
8 changes: 8 additions & 0 deletions pkg/cloud/services/network/routetables.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,13 @@ func (s *Service) getNatGatewayPrivateRoute(natGatewayID string) *ec2.CreateRout
}
}

func (s *Service) getNat64PrivateRoute(natGatewayID string) *ec2.CreateRouteInput {
return &ec2.CreateRouteInput{
NatGatewayId: aws.String(natGatewayID),
DestinationIpv6CidrBlock: aws.String(services.NAT64CidrBlock),
}
}

func (s *Service) getEgressOnlyInternetGateway() *ec2.CreateRouteInput {
return &ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String(services.AnyIPv6CidrBlock),
Expand Down Expand Up @@ -415,6 +422,7 @@ func (s *Service) getRoutesToPrivateSubnet(sn *infrav1.SubnetSpec) (routes []*ec

routes = append(routes, s.getNatGatewayPrivateRoute(natGatewayID))
if sn.IsIPv6 {
routes = append(routes, s.getNat64PrivateRoute(natGatewayID))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Not sure why this was not needed for managed installs?
  2. Do we need to check for !sn.IsPublic here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check for !sn.IsPublic here?

I think the check is already done in

cluster-api-provider-aws/pkg/cloud/services/network/routetables.go

Lines 429 to 434 in 6f867be

func (s *Service) getRoutesForSubnet(sn *infrav1.SubnetSpec) ([]*ec2.CreateRouteInput, error) {
if sn.IsPublic {
return s.getRoutesToPublicSubnet(sn)
}
return s.getRoutesToPrivateSubnet(sn)
}

The func getRoutesToPrivateSubnet is only ever used for getting routes for private subnets :D

Not sure why this was not needed for managed installs

This route is actually a component of DNS64 (NAT64 is always on) to resolve an IPv4-only internet service to a synthetic IPv6 (See here). It is really a nice-to-have feature 🤔 but not required.

My guess was that when Ipv6 was supported for managed eks, they didn't need IPv6-only pods to access IPv4-only internet services.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, I think CAPA currently support IPv6 for EKS by using dualstack subnets. And NAT64/DNS64 is really meant for IPv6-only subnets, not dualstack subnets (unless really necessary).

See details in commit: 78cb9d4

if !s.scope.VPC().IsIPv6Enabled() {
// Safety net because EgressOnlyInternetGateway needs the ID from the ipv6 block.
// if, for whatever reason by this point that is not available, we don't want to
Expand Down
18 changes: 18 additions & 0 deletions pkg/cloud/services/network/routetables_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,13 @@ func TestReconcileRouteTables(t *testing.T) {
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("64:ff9b::/96"),
NatGatewayId: aws.String("nat-01"),
RouteTableId: aws.String("rt-1"),
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("::/0"),
EgressOnlyInternetGatewayId: aws.String("eigw-01"),
Expand Down Expand Up @@ -247,6 +254,13 @@ func TestReconcileRouteTables(t *testing.T) {
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("64:ff9b::/96"),
NatGatewayId: aws.String("nat-01"),
RouteTableId: aws.String("rt-1"),
})).
After(privateRouteTable)

m.CreateRoute(context.TODO(), gomock.Eq(&ec2.CreateRouteInput{
DestinationIpv6CidrBlock: aws.String("::/0"),
EgressOnlyInternetGatewayId: aws.String("eigw-01"),
Expand Down Expand Up @@ -1199,6 +1213,10 @@ func TestService_getRoutesForSubnet(t *testing.T) {
DestinationCidrBlock: aws.String("0.0.0.0/0"),
NatGatewayId: aws.String("nat-gw-fromZone-us-east-1a"),
},
{
DestinationIpv6CidrBlock: aws.String("64:ff9b::/96"),
NatGatewayId: aws.String("nat-gw-fromZone-us-east-1a"),
},
{
DestinationIpv6CidrBlock: aws.String("::/0"),
EgressOnlyInternetGatewayId: aws.String("vpc-eigw"),
Expand Down
22 changes: 22 additions & 0 deletions pkg/cloud/services/network/subnets.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,6 +545,28 @@ func (s *Service) createSubnet(sn *infrav1.SubnetSpec) (*infrav1.SubnetSpec, err
return nil, errors.Wrapf(err, "failed to set subnet %q attribute assign ipv6 address on creation", *out.Subnet.SubnetId)
}
record.Eventf(s.scope.InfraCluster(), "SuccessfulModifySubnetAttributes", "Modified managed Subnet %q attributes", *out.Subnet.SubnetId)

// Enable DNS64 so that the Route 53 Resolver returns DNS records for IPv4-only services
// containing a synthesized IPv6 address prefixed 64:ff9b::/96.
// This is needed alongside NAT64 to allow IPv6-only workloads to reach IPv4-only services.
// We only need to enable on private subnets.
if !sn.IsPublic {
if err := wait.WaitForWithRetryable(wait.NewBackoff(), func() (bool, error) {
if _, err := s.EC2Client.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
SubnetId: out.Subnet.SubnetId,
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
}); err != nil {
return false, err
}
return true, nil
}, awserrors.SubnetNotFound); err != nil {
record.Warnf(s.scope.InfraCluster(), "FailedModifySubnetAttributes", "Failed modifying managed Subnet %q attributes: %v", *out.Subnet.SubnetId, err)
return nil, errors.Wrapf(err, "failed to set subnet %q attribute enable dns64", *out.Subnet.SubnetId)
}
record.Eventf(s.scope.InfraCluster(), "SuccessfulModifySubnetAttributes", "Modified managed Subnet %q attributes", *out.Subnet.SubnetId)
}
}

// AWS Wavelength Zone's public subnets does not support to map Carrier IP address on launch, and
Expand Down
102 changes: 68 additions & 34 deletions pkg/cloud/services/network/subnets_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1781,14 +1781,6 @@ func TestReconcileSubnets(t *testing.T) {
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -1865,6 +1857,22 @@ func TestReconcileSubnets(t *testing.T) {
}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.DescribeAvailabilityZones(context.TODO(), gomock.Any()).
Return(&ec2.DescribeAvailabilityZonesOutput{
AvailabilityZones: []types.AvailabilityZone{
Expand Down Expand Up @@ -3656,15 +3664,6 @@ func TestReconcileSubnets(t *testing.T) {
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(firstSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -3741,6 +3740,22 @@ func TestReconcileSubnets(t *testing.T) {
}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(secondSubnet)

m.DescribeAvailabilityZones(context.TODO(), gomock.Any()).
Return(&ec2.DescribeAvailabilityZonesOutput{
AvailabilityZones: []types.AvailabilityZone{
Expand Down Expand Up @@ -3903,15 +3918,6 @@ func TestReconcileSubnets(t *testing.T) {
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PublicSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PublicSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -3988,6 +3994,24 @@ func TestReconcileSubnets(t *testing.T) {
}, nil).
After(zone1PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone1PrivateSubnet)

// zone 2
m.DescribeAvailabilityZones(context.TODO(), &ec2.DescribeAvailabilityZonesInput{
ZoneNames: []string{"us-east-1c"},
Expand Down Expand Up @@ -4077,14 +4101,6 @@ func TestReconcileSubnets(t *testing.T) {
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PublicSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PublicSubnet)
m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
MapPublicIpOnLaunch: &types.AttributeBooleanValue{
Value: aws.Bool(true),
Expand Down Expand Up @@ -4160,6 +4176,24 @@ func TestReconcileSubnets(t *testing.T) {
},
}, nil).
After(zone2PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
AssignIpv6AddressOnCreation: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PrivateSubnet)

m.ModifySubnetAttribute(context.TODO(), &ec2.ModifySubnetAttributeInput{
EnableDns64: &types.AttributeBooleanValue{
Value: aws.Bool(true),
},
SubnetId: aws.String("subnet-2"),
}).
Return(&ec2.ModifySubnetAttributeOutput{}, nil).
After(zone2PrivateSubnet)
},
},
}
Expand Down