Skip to content

✨ IPv6 support for self-managed clusters #5603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
+33,480 −5,671
Open

✨ IPv6 support for self-managed clusters #5603

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
7949ccb
validations: allow IPv6 configurations for unmanaged clusters
tthvo Jul 22, 2025
c1bafb0
ec2: enable primary IPv6 on ENI for EC2 instances
tthvo Jul 22, 2025
842f85a
ec2: support option HTTPProtocolIPv6 for EC2 IMDS
tthvo Jul 22, 2025
60308b2
routing: ensure routes to eigw are up to date
tthvo Jul 22, 2025
7826e7e
subnets: configure default subnets to use NAT64/DNS64
tthvo Jul 23, 2025
987bc9c
securitygroup: ensure icmpv6 is supported
tthvo Jul 23, 2025
7bbae60
securitygroup: allow setting allowed IPv6 CIDR for node NodePort serv…
tthvo Jul 28, 2025
cd6c178
securitygroup: allow configuring IPv6 source CIDRs for bastion SSH
tthvo Jul 28, 2025
1ebffda
crd: add IPv6 of bastion host to cluster status
tthvo Jul 30, 2025
2c7e1e4
template: manifest templates for IPv6-enabled cluster
tthvo Jul 29, 2025
da96854
cni: customized calico manifests for single-stack IPv6
tthvo Jul 29, 2025
cd7c9b2
docs: add documentations for enabling IPv6 in non-eks clusters
tthvo Jul 29, 2025
def75f2
validations: validate vpc and subnet CIDR
tthvo Aug 5, 2025
9851222
docs: update doc for enabling ipv6
tthvo Aug 6, 2025
ea41d67
cni: document the requirement for calico ipv6 support
tthvo Aug 8, 2025
bfbbb84
subnets: wait till IPv6 CIDR is associated with subnets
tthvo Sep 19, 2025
4d38ed3
sg: allow both ipv4 and ipv6 cidrs to API LB if vpc ipv6 block is def…
tthvo Sep 29, 2025
e5e3ad3
crd: clarify isIpv6 field on subnet spec
tthvo Jul 29, 2025
04d4286
api: add spec field to configure target group ipType
tthvo Oct 2, 2025
b8ca23a
subnets: auto-assign IPv6 CIDR blocks to subnets when not specified
tthvo Oct 6, 2025
3292f75
vpc: ipam pool under vpc.ipv6 should be used for VPC IPv6 CIDR
tthvo Oct 9, 2025
78cb9d4
subnets: only enable DNS64 for IPv6-only subnets
tthvo Oct 10, 2025
061134a
docs: add dualstack cluster support documentation
tthvo Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions api/v1beta1/network_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,6 @@ type SubnetSpec struct {

// IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
// A subnet can have an IPv4 and an IPv6 address.
// IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
// +optional
IPv6CidrBlock string `json:"ipv6CidrBlock,omitempty"`

Expand All @@ -261,7 +260,6 @@ type SubnetSpec struct {
IsPublic bool `json:"isPublic"`

// IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
// IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is possible to have an IPv6-enabled VPC that contains IPv4 subnets. In that case, I would assume that this field would be unset or explicitly set to false i.e even of IPv6 enabled VPCs, IPV4 subnets would be the default? Could you please confirm if that is the case and also update description to reflect what the default would be?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far, I only consider the "happy" default path that vpc and subnets are dual-stack. Let me add this to my list of questions to confirm. Will get back asap.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I updated the CRD description to:

IsIPv6 defines the subnet as an IPv6 subnet.
A subnet is IPv6 when it is associated with an IPv6 CIDR.

This should mean that IsIPv6 reflects the state of the subnet (i.e. not depending on the VPC). So, an Ipv4 subnet will have sn.IsIPv6==false as expected even in a dualstack VPC.

Commit: 43d6ec8

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, CAPA will describe the subnet and update the field to reflect the correct state (i.e. IPv6 or not, depending on whether there is an associated IPv6 CIDR)

Describing subnets:

cluster-api-provider-aws/pkg/cloud/services/network/subnets.go

Lines 405 to 411 in f3c2166

spec.CidrBlock = aws.ToString(ec2sn.CidrBlock)
for _, set := range ec2sn.Ipv6CidrBlockAssociationSet {
if set.Ipv6CidrBlockState.State == types.SubnetCidrBlockStateCodeAssociated {
spec.IPv6CidrBlock = aws.ToString(set.Ipv6CidrBlock)
spec.IsIPv6 = true
}
}

Deep-copy subnet state to spec:

cluster-api-provider-aws/pkg/cloud/services/network/subnets.go

Lines 136 to 137 in f3c2166

// Update subnet spec with the existing subnet details
existingSubnet.DeepCopyInto(sub)

// +optional
IsIPv6 bool `json:"isIpv6,omitempty"`

Expand Down
6 changes: 0 additions & 6 deletions api/v1beta2/awscluster_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,13 +301,7 @@ func (r *AWSCluster) validateSSHKeyName() field.ErrorList {

func (r *AWSCluster) validateNetwork() field.ErrorList {
var allErrs field.ErrorList
if r.Spec.NetworkSpec.VPC.IsIPv6Enabled() {
allErrs = append(allErrs, field.Invalid(field.NewPath("ipv6"), r.Spec.NetworkSpec.VPC.IPv6, "IPv6 cannot be used with unmanaged clusters at this time."))
}
for _, subnet := range r.Spec.NetworkSpec.Subnets {
if subnet.IsIPv6 || subnet.IPv6CidrBlock != "" {
allErrs = append(allErrs, field.Invalid(field.NewPath("subnets"), r.Spec.NetworkSpec.Subnets, "IPv6 cannot be used with unmanaged clusters at this time."))
}
if subnet.ZoneType != nil && subnet.IsEdge() {
if subnet.ParentZoneName == nil {
allErrs = append(allErrs, field.Invalid(field.NewPath("subnets"), r.Spec.NetworkSpec.Subnets, "ParentZoneName must be set when ZoneType is 'local-zone'."))
Expand Down
12 changes: 6 additions & 6 deletions api/v1beta2/awscluster_webhook_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ func TestAWSClusterValidateCreate(t *testing.T) {
wantErr: false,
},
{
name: "rejects ipv6",
name: "accepts ipv6",
cluster: &AWSCluster{
Spec: AWSClusterSpec{
NetworkSpec: NetworkSpec{
Expand All @@ -337,10 +337,10 @@ func TestAWSClusterValidateCreate(t *testing.T) {
},
},
},
wantErr: true,
wantErr: false,
},
{
name: "rejects ipv6 enabled subnet",
name: "accepts ipv6 enabled subnet",
cluster: &AWSCluster{
Spec: AWSClusterSpec{
NetworkSpec: NetworkSpec{
Expand All @@ -356,10 +356,10 @@ func TestAWSClusterValidateCreate(t *testing.T) {
},
},
},
wantErr: true,
wantErr: false,
},
{
name: "rejects ipv6 cidr block for subnets",
name: "accepts ipv6 cidr block for subnets",
cluster: &AWSCluster{
Spec: AWSClusterSpec{
NetworkSpec: NetworkSpec{
Expand All @@ -372,7 +372,7 @@ func TestAWSClusterValidateCreate(t *testing.T) {
},
},
},
wantErr: true,
wantErr: false,
},
{
name: "rejects ingress rules with cidr block and source security group id",
Expand Down
1 change: 1 addition & 0 deletions api/v1beta2/awsmachinetemplate_webhook_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ func TestAWSMachineTemplateValidateUpdate(t *testing.T) {
InstanceType: "test",
InstanceMetadataOptions: &InstanceMetadataOptions{
HTTPEndpoint: InstanceMetadataEndpointStateEnabled,
HTTPProtocolIPv6: InstanceMetadataEndpointStateDisabled,
HTTPPutResponseHopLimit: 1,
HTTPTokens: HTTPTokensStateOptional,
InstanceMetadataTags: InstanceMetadataEndpointStateDisabled,
Expand Down
5 changes: 1 addition & 4 deletions api/v1beta2/network_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,8 +432,7 @@ type VPCSpec struct {
// Mutually exclusive with CidrBlock.
IPAMPool *IPAMPool `json:"ipamPool,omitempty"`

// IPv6 contains ipv6 specific settings for the network. Supported only in managed clusters.
// This field cannot be set on AWSCluster object.
// IPv6 contains ipv6 specific settings for the network.
// +optional
IPv6 *IPv6 `json:"ipv6,omitempty"`

Expand Down Expand Up @@ -561,7 +560,6 @@ type SubnetSpec struct {

// IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
// A subnet can have an IPv4 and an IPv6 address.
// IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
// +optional
IPv6CidrBlock string `json:"ipv6CidrBlock,omitempty"`

Expand All @@ -573,7 +571,6 @@ type SubnetSpec struct {
IsPublic bool `json:"isPublic"`

// IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
// IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
// +optional
IsIPv6 bool `json:"isIpv6,omitempty"`

Expand Down
12 changes: 12 additions & 0 deletions api/v1beta2/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,15 @@ type InstanceMetadataOptions struct {
// +kubebuilder:default=enabled
HTTPEndpoint InstanceMetadataState `json:"httpEndpoint,omitempty"`

// Enables or disables the IPv6 endpoint for the instance metadata service.
// This applies only if you enabled the HTTP metadata endpoint.
//
// Default: disabled
//
// +kubebuilder:validation:Enum:=enabled;disabled
// +kubebuilder:default=disabled
HTTPProtocolIPv6 InstanceMetadataState `json:"httpProtocolIpv6,omitempty"`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Should this be HTTPIPv6Endpoint or HTTPEndpointIPv6 to be consistent with HTTPEndpoint for IPv4?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, the name HTTPProtocolIPv6 is actually used to match the corresponding AWS metadata option field, for example:

$ aws ec2 modify-instance-metadata-options --instance-id i-1234567890 \
    --http-endpoint=disabled --http-protocol-ipv6=enabled
{
    "InstanceId": "i-1234567890",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 2,
        "HttpEndpoint": "disabled",
        "HttpProtocolIpv6": "enabled",
        "InstanceMetadataTags": "disabled"
    }
}

https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-metadata-options.html

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's the convention here in capa 👀 so we should keep it this way?


// The desired HTTP PUT response hop limit for instance metadata requests. The
// larger the number, the further instance metadata requests can travel.
//
Expand Down Expand Up @@ -411,6 +420,9 @@ func (obj *InstanceMetadataOptions) SetDefaults() {
if obj.HTTPEndpoint == "" {
obj.HTTPEndpoint = InstanceMetadataEndpointStateEnabled
}
if obj.HTTPProtocolIPv6 == "" {
obj.HTTPProtocolIPv6 = InstanceMetadataEndpointStateDisabled
}
if obj.HTTPPutResponseHopLimit == 0 {
obj.HTTPPutResponseHopLimit = 1
}
Expand Down
46 changes: 32 additions & 14 deletions config/crd/bases/controlplane.cluster.x-k8s.io_awsmanagedcontrolplanes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,12 +599,11 @@ spec:
description: |-
IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
A subnet can have an IPv4 and an IPv6 address.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
type: string
isIpv6:
description: |-
IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
description: IsIPv6 defines the subnet as an IPv6 subnet.
A subnet is IPv6 when it is associated with a VPC that
has IPv6 enabled.
type: boolean
isPublic:
description: IsPublic defines the subnet as a public subnet.
Expand Down Expand Up @@ -785,9 +784,8 @@ spec:
type: integer
type: object
ipv6:
description: |-
IPv6 contains ipv6 specific settings for the network. Supported only in managed clusters.
This field cannot be set on AWSCluster object.
description: IPv6 contains ipv6 specific settings for the
network.
properties:
cidrBlock:
description: |-
Expand Down Expand Up @@ -1309,6 +1307,17 @@ spec:
- enabled
- disabled
type: string
httpProtocolIpv6:
default: disabled
description: |-
Enables or disables the IPv6 endpoint for the instance metadata service.
This applies only if you enabled the HTTP metadata endpoint.

Default: disabled
enum:
- enabled
- disabled
type: string
httpPutResponseHopLimit:
default: 1
description: |-
Expand Down Expand Up @@ -2823,12 +2832,11 @@ spec:
description: |-
IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
A subnet can have an IPv4 and an IPv6 address.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
type: string
isIpv6:
description: |-
IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
description: IsIPv6 defines the subnet as an IPv6 subnet.
A subnet is IPv6 when it is associated with a VPC that
has IPv6 enabled.
type: boolean
isPublic:
description: IsPublic defines the subnet as a public subnet.
Expand Down Expand Up @@ -3009,9 +3017,8 @@ spec:
type: integer
type: object
ipv6:
description: |-
IPv6 contains ipv6 specific settings for the network. Supported only in managed clusters.
This field cannot be set on AWSCluster object.
description: IPv6 contains ipv6 specific settings for the
network.
properties:
cidrBlock:
description: |-
Expand Down Expand Up @@ -3586,6 +3593,17 @@ spec:
- enabled
- disabled
type: string
httpProtocolIpv6:
default: disabled
description: |-
Enables or disables the IPv6 endpoint for the instance metadata service.
This applies only if you enabled the HTTP metadata endpoint.

Default: disabled
enum:
- enabled
- disabled
type: string
httpPutResponseHopLimit:
default: 1
description: |-
Expand Down
30 changes: 19 additions & 11 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,12 +284,11 @@ spec:
description: |-
IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
A subnet can have an IPv4 and an IPv6 address.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
type: string
isIpv6:
description: |-
IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
description: IsIPv6 defines the subnet as an IPv6 subnet.
A subnet is IPv6 when it is associated with a VPC that
has IPv6 enabled.
type: boolean
isPublic:
description: IsPublic defines the subnet as a public subnet.
Expand Down Expand Up @@ -1546,12 +1545,11 @@ spec:
description: |-
IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
A subnet can have an IPv4 and an IPv6 address.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
type: string
isIpv6:
description: |-
IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
description: IsIPv6 defines the subnet as an IPv6 subnet.
A subnet is IPv6 when it is associated with a VPC that
has IPv6 enabled.
type: boolean
isPublic:
description: IsPublic defines the subnet as a public subnet.
Expand Down Expand Up @@ -1732,9 +1730,8 @@ spec:
type: integer
type: object
ipv6:
description: |-
IPv6 contains ipv6 specific settings for the network. Supported only in managed clusters.
This field cannot be set on AWSCluster object.
description: IPv6 contains ipv6 specific settings for the
network.
properties:
cidrBlock:
description: |-
Expand Down Expand Up @@ -2287,6 +2284,17 @@ spec:
- enabled
- disabled
type: string
httpProtocolIpv6:
default: disabled
description: |-
Enables or disables the IPv6 endpoint for the instance metadata service.
This applies only if you enabled the HTTP metadata endpoint.

Default: disabled
enum:
- enabled
- disabled
type: string
httpPutResponseHopLimit:
default: 1
description: |-
Expand Down
19 changes: 8 additions & 11 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsclustertemplates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,12 +303,11 @@ spec:
description: |-
IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
A subnet can have an IPv4 and an IPv6 address.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
type: string
isIpv6:
description: |-
IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
description: IsIPv6 defines the subnet as an IPv6
subnet. A subnet is IPv6 when it is associated
with a VPC that has IPv6 enabled.
type: boolean
isPublic:
description: IsPublic defines the subnet as a public
Expand Down Expand Up @@ -1133,12 +1132,11 @@ spec:
description: |-
IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
A subnet can have an IPv4 and an IPv6 address.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
type: string
isIpv6:
description: |-
IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
description: IsIPv6 defines the subnet as an IPv6
subnet. A subnet is IPv6 when it is associated
with a VPC that has IPv6 enabled.
type: boolean
isPublic:
description: IsPublic defines the subnet as a public
Expand Down Expand Up @@ -1320,9 +1318,8 @@ spec:
type: integer
type: object
ipv6:
description: |-
IPv6 contains ipv6 specific settings for the network. Supported only in managed clusters.
This field cannot be set on AWSCluster object.
description: IPv6 contains ipv6 specific settings
for the network.
properties:
cidrBlock:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsmachinepools.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -707,6 +707,17 @@ spec:
- enabled
- disabled
type: string
httpProtocolIpv6:
default: disabled
description: |-
Enables or disables the IPv6 endpoint for the instance metadata service.
This applies only if you enabled the HTTP metadata endpoint.

Default: disabled
enum:
- enabled
- disabled
type: string
httpPutResponseHopLimit:
default: 1
description: |-
Expand Down
11 changes: 11 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsmachines.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -902,6 +902,17 @@ spec:
- enabled
- disabled
type: string
httpProtocolIpv6:
default: disabled
description: |-
Enables or disables the IPv6 endpoint for the instance metadata service.
This applies only if you enabled the HTTP metadata endpoint.

Default: disabled
enum:
- enabled
- disabled
type: string
httpPutResponseHopLimit:
default: 1
description: |-
Expand Down
11 changes: 11 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsmachinetemplates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -821,6 +821,17 @@ spec:
- enabled
- disabled
type: string
httpProtocolIpv6:
default: disabled
description: |-
Enables or disables the IPv6 endpoint for the instance metadata service.
This applies only if you enabled the HTTP metadata endpoint.
Default: disabled
enum:
- enabled
- disabled
type: string
httpPutResponseHopLimit:
default: 1
description: |-
Expand Down
Loading