WIP: Add support for dual stack load balancers #1313

nrb · 2025-12-02T21:18:23Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

Adds basic support for configuring the IP address family on Services.

This functionality overlaps with the ALBC somewhat, but the goal here is to provide much more basic functionality, not the full scope of ALBC.

Which issue(s) this PR fixes:

Fixes #1219

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Introduce `service.beta.kubernetes.io/aws-load-balancer-ip-address-type` to enable dual stack Services.

k8s-ci-robot · 2025-12-02T21:18:26Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-12-02T21:18:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign cartermckinnon for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-12-02T21:18:31Z

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nrb · 2025-12-02T21:32:24Z

/test pull-cloud-provider-aws-test

damdo · 2025-12-09T19:49:55Z

Hey @kmala do we know what's happening with the Netlify checks? It looks like they are broken across many open PRs. TY

damdo · 2025-12-10T07:50:27Z

/retest

nrb · 2025-12-11T16:00:12Z

/retest

nrb · 2025-12-12T14:00:46Z

/retest

nrb · 2025-12-12T16:06:24Z

/test pull-cloud-provider-aws-test

One more run to see if this is a rate limiting issue or some sort of genuine issue with my code.

tthvo

I tested with a simple service below:

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
spec:
  ipFamilyPolicy: PreferDualStack
  ports:
    - port: 3306
  selector:
    app: wordpress
  type: LoadBalancer

The CCM does create the dualstack NLB as expected 🚀

$ kubectl -n myns get svc/wordpress-mysql -o yaml | yq .status.loadBalancer.ingress
- hostname: <id>.elb.us-east-1.amazonaws.com

$ nslookup <id>.elb.us-east-1.amazonaws.com
# nslookup shows both public IPv4 EIP and IPv6 GUA addresses

$ aws elbv2 describe-load-balancers --names=<nlb-id> --query='LoadBalancers[0].[Type,IpAddressType,Scheme]'
[
    "network",
    "dualstack",
    "internet-facing"
]

pkg/providers/v1/aws_validations_test.go

pkg/providers/v1/aws.go

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb · 2026-01-16T15:08:12Z

/test pull-cloud-provider-aws-test

Add support for annotating a Service with a desired IP address type. As an example: apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" service.beta.kubernetes.io/aws-load-balancer-ip-address-type: "dualstack" service.beta.kubernetes.io/aws-load-balancer-target-group-ip-address-type: "ipv6" spec: type: LoadBalancer Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

The Kubernetes Service API allows load balancer IP families to be configured via the Service.spec.ipFamilies and Service.spec.ipFamilyPolicy fields. These should be considered along with the annotations that we are adding, especially since this is how other cloud providers do it. Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb · 2026-01-22T22:41:32Z

/test pull-cloud-provider-aws-test

nrb · 2026-01-22T22:42:51Z

/test pull-cloud-provider-aws-check

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb · 2026-01-23T17:38:16Z

/test pull-cloud-provider-aws-test
/test pull-cloud-provider-aws-check

nrb · 2026-01-26T15:29:02Z

@kmala I'm going to do some manual testing with this PR starting today before I mark it as ready for review, but would you be able to give it a look by any chance?

nrb · 2026-01-26T15:29:11Z

/assign @kmala

nrb · 2026-01-28T21:03:22Z

Tagging you all because we've talked at various points.

@tthvo (also working on dual stack enablement in Cluster API Provider AWS)
@mtulio (also helping with dual stack tasks)
@kmala (AWS CCM maintainer)
@alebedev87 (ingress controller SME)

Summing up some of the various discussions. Please correct me if any of these things are wrong.

https://kubernetes.io/docs/concepts/services-networking/dual-stack/ talks about an extension to the Service resource to specify IP family information. For our purposes, it allows us to specify...
- Precedence of IPv4 or IPv6 addresses depending on their order in the spec.ipFamilies field
- How to handle failure on dual stack LBs - fall back to IPv4/single stack, or fail creation call
What this API does not give us, compared to using annotations
- No way to specify attachment type, whether ip or instance; that is, what backend the LB will route to - the instance, even with a new IP, or the IP, even if it's a new instance.
- Difficult or unclear on how to specify the IP family on a target group; a dual stack LB could use an IPv4 target group or an IPv6 target group, but not both. As-is, we'd need to infer it based on the order of spec.ipFamilies.
- Similar issues with security groups; no way to specify them, but we could infer based on spec.ipFamilies

I can see us proceeding in a few ways here.

Keep the API in this repo fairly simple; infer information based on spec.ipFamilies and spec.ipFamilyPolicy fields, and do not use any annotations
- Security/target groups will be created based on whichever IP family is first in spec.ipFamilies
- Use instance attachment since that's how the CCM has behaved so far.
- This aligns us with the upstream API and keeps the implementation simple, but at the cost of reduced flexibility
Use only annotations to specify this information, the same way AWS Load Balancer Controller does. Use the Service.spec.LoadBalancerClass to determine if the CCM should process the Service or the ALBC should.
- This is beneficial because the user or higher level component can explicitly and fully communicate settings.
- This might be challenging with the Gateway API, which can only propagate 8 annotations
- Maximal flexibility, but annotations as an API is difficult to fully validate and may not be fully supported by higher-level controllers.
Use a combination of the Service spec and annotations; service.spec.ipFamilies and service.spec.ipFamilyPolicy would be used for their respective purposes, and further settings, like security/target groups or attachment type, are specified via annotations.
- Combining both approaches allows us to present a small required API, but still offer customization for users that need it.
- This includes the downsides of both approaches, plus it adds mental overhead in understanding what information goes where.

In any of the 3 approaches here, I'd like to keep the number of total fields to the absolute minimum for supporting dual stack on load balancers. If users need something beyond that, they should opt for the ALBC.

That minimum would be:

Fallback behavior (Service.spec.ipFamilyPolicy fulfills this)
Supported IP families and priority ('Service.spec.ipFamilies' fulfills this)
Target group IP family (could be inferred/defaulted)
Security group IP family (could be inferred/defaulted)
Instance attachment type (could be inferred/defaulted)

I'd appreciate your thoughts here - which of these options seems more workable to you?

mtulio

Great work! A few nits, and questions related to the restrictions on target ipv6.

mtulio · 2026-01-28T21:08:06Z

pkg/providers/v1/aws.go

+// ServiceAnnotationLoadBalancerIPAddressType is the annotation used on the service
+// to specify the IP address type for the load balancer. Supported values are "ipv4" and "dualstack".
+// Defaults to "ipv4". Only supported on NLB.
+const ServiceAnnotationLoadBalancerIPAddressType = "service.beta.kubernetes.io/aws-load-balancer-ip-address-type"
+
+// ServiceAnnotationLoadBalancerTargetGroupIPAddressType is the annotation used on the service
+// to specify the IP address type for the target groups. Supported values are "ipv4" and "ipv6".
+// Defaults to "ipv4". Only supported on NLB.
+const ServiceAnnotationLoadBalancerTargetGroupIPAddressType = "service.beta.kubernetes.io/aws-load-balancer-target-group-ip-address-type"
+


Please also update the service controller documentation: https://github.com/kubernetes/cloud-provider-aws/blob/master/docs/service_controller.md

mtulio · 2026-01-28T21:12:35Z

pkg/providers/v1/aws.go

 		if len(sourceRangeCidrs) == 0 {
 			sourceRangeCidrs = append(sourceRangeCidrs, "0.0.0.0/0")
+
+			// For dual-stack or IPv6 load balancers, also add IPv6 default route


route or rules?

Suggested change

// For dual-stack or IPv6 load balancers, also add IPv6 default route

// For dual-stack or IPv6 load balancers, also add IPv6 default rules

mtulio · 2026-01-28T21:24:28Z

pkg/providers/v1/aws_validations.go

+	// Validate ipFamilyPolicy and ipFamilies
+	if v.apiService.Spec.IPFamilyPolicy != nil {


nit: what about return early when IPFamilyPolicy==nil?

mtulio · 2026-01-28T21:46:18Z

pkg/providers/v1/aws_validations.go

+	if _, present := v.annotations[ServiceAnnotationLoadBalancerIPAddressType]; present {
+		if !isNLB {
+			return fmt.Errorf("ip address type annotation is only supported for NLB")
+		}
+	}
+
+	if _, present := v.annotations[ServiceAnnotationLoadBalancerTargetGroupIPAddressType]; present {
+		if !isNLB {
+			return fmt.Errorf("target group ip address type annotation is only supported for NLB")
+		}
+	}


do we need to validate valid values were set here?

mtulio · 2026-01-28T21:58:11Z

pkg/providers/v1/aws_loadbalancer.go

+	}
+
+	// Fall back to spec.ipFamilies and spec.ipFamilyPolicy
+	if service.Spec.IPFamilyPolicy != nil {


what about return early here with elbv2types.IpAddressTypeIpv4?

mtulio · 2026-01-28T22:35:13Z

pkg/providers/v1/aws_validations.go

+	if _, present := v.annotations[ServiceAnnotationLoadBalancerIPAddressType]; present {
+		if !isNLB {
+			return fmt.Errorf("ip address type annotation is only supported for NLB")
+		}


Considering this statement from AWS docs:

Also, with dualstack Network Load Balancers, client IP address preservation does not work when translating IPv4 addresses to IPv6, or IPv6 to IPv4 addresses. Client IP address preservation only works when client and target IP addresses are both IPv4 or both IPv6.

Do we need to add some validation when the preserve ip is set?

mtulio · 2026-01-28T22:37:22Z

pkg/providers/v1/aws_loadbalancer.go

+		}
+		if len(networkInterface.Ipv6Addresses) > 0 {
+			// Return the first IPv6 address
+			return aws.ToString(networkInterface.Ipv6Addresses[0].Ipv6Address)


In order to satisfy the requirement of the target ip family ipv6, I believe you need to verify if the Ipv6Address is primary (IsPrimaryIpv6: true).

Thoughts @tthvo @nrb ?

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_NetworkInterfaceIpv6Address.html

When registering targets by instance ID for a IPv6 target group, the targets must have an assigned primary IPv6 address. To learn more, see IPv6 addresses in the Amazon EC2 User Guide

When registering targets by IP address for an IPv6 target group, the IP addresses that you register must be within the VPC IPv6 CIDR block or within the IPv6 CIDR block of a peered VPC.

Hey @mtulio That's right, but the primary IPv6 requirement actually depends on the target type of the Target Group. In our case, the following type is applicable:

instance: The instance must be in the same VPC as NLB and is assigned a "stable" IPv6 address. Unlike IPv4, IPv6 address, by default, can be re-assigned elsewhere. Thus, we need to set IsPrimaryIpv6: true to make sure the IPv6 address is tied to the ENI of the instance for its entire lifecycle. I guess "primary" on AWS means "stable/owned".

When registering targets by instance ID for a IPv6 target group, the targets must have an assigned primary IPv6 address. To learn more, see IPv6 addresses in the Amazon EC2 User Guide

IP: There is no requirement for a "stable" IPv6 address (or setting flag IsPrimaryIpv6). But the IPv6 address must be within the VPC IPv6 CIDR of the cluster.

When registering targets by IP address for an IPv6 target group, the IP addresses that you register must be within the VPC IPv6 CIDR block or within the IPv6 CIDR block of a peered VPC.

My understanding is that we should prefer instanceID target type in order to ensure traffic will always reach the intended target (i.e. cluster nodes). Registering with IP address will pin the traffic to the IP, which may not be the cluster node at all as it can be re-assigned to other services, for example, another non-cluster EC2 instance.

@nrb IIUC, we should make the CCM prefer registering by instance when, let's say, primary IPv6 is available. Otherwise, fall back to register by the IP address?

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Dec 2, 2025

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 2, 2025

k8s-ci-robot requested review from bridgetkromhout and cheftako December 2, 2025 21:18

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 2, 2025

nrb force-pushed the dual-stack-lb branch from ba4b93b to 83110d8 Compare December 11, 2025 19:04

tthvo reviewed Jan 7, 2026

View reviewed changes

pkg/providers/v1/aws_validations_test.go Outdated Show resolved Hide resolved

pkg/providers/v1/aws.go Show resolved Hide resolved

nrb added 2 commits January 14, 2026 14:05

Add support for dual stack load balancers

70fa363

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Add test to confirm dual stack can't be on CLBs

ffbb4b8

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb force-pushed the dual-stack-lb branch from 83110d8 to ffbb4b8 Compare January 16, 2026 13:58

nrb added 2 commits January 20, 2026 14:32

Add end to end test for dualstack NLBs

d2315dd

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 20, 2026

nrb added 3 commits January 21, 2026 15:30

Register nodes with IPv6 for IPv6 target group

de00cc6

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Use IPv6 security groups with dual stack LBs

713764e

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Add fakes and e2e test for IPv6 SGs

73d10dc

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 21, 2026

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 22, 2026

nrb added 2 commits January 23, 2026 08:57

Formatting fixes

5652364

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Mock security group calls for integration tests

4d5b485

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

k8s-ci-robot assigned kmala Jan 26, 2026

mtulio reviewed Jan 28, 2026

View reviewed changes

	// For dual-stack or IPv6 load balancers, also add IPv6 default route
	// For dual-stack or IPv6 load balancers, also add IPv6 default rules

		// Validate ipFamilyPolicy and ipFamilies
		if v.apiService.Spec.IPFamilyPolicy != nil {

WIP: Add support for dual stack load balancers #1313

Are you sure you want to change the base?

WIP: Add support for dual stack load balancers #1313

Uh oh!

Conversation

nrb commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 2, 2025

Uh oh!

nrb commented Dec 2, 2025

Uh oh!

damdo commented Dec 9, 2025

Uh oh!

damdo commented Dec 10, 2025

Uh oh!

nrb commented Dec 11, 2025

Uh oh!

nrb commented Dec 12, 2025

Uh oh!

nrb commented Dec 12, 2025

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nrb commented Jan 16, 2026

Uh oh!

nrb commented Jan 22, 2026

Uh oh!

nrb commented Jan 22, 2026

Uh oh!

nrb commented Jan 23, 2026

Uh oh!

nrb commented Jan 26, 2026

Uh oh!

nrb commented Jan 26, 2026

Uh oh!

nrb commented Jan 28, 2026

Uh oh!

mtulio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants