roachprod: add support for create/list/destroy load balancer (AWS)

[cpj2195]

This change adds Network Load Balancer (NLB) support for AWS clusters by
implementing the CreateLoadBalancer, DeleteLoadBalancer, and
ListLoadBalancers methods on the AWS provider.

The implementation:

• Creates a regional NLB with a TCP listener on the specified port
• Creates a target group with health checks and registers all cluster VMs
• Uses a naming convention {cluster}-{port}-{type}-roachprod (truncated to
  32 chars due to AWS limits)
• Handles multi-region clusters by creating an NLB in each region
• Cleans up all associated resources (listeners, target groups) on deletion
• Supports deleting all load balancers for a cluster

Usage

Create a load balancer for SQL connections

roachprod load-balancer create <$cluster> --secure

List load balancers and view connection info

roachprod load-balancer list <$cluster>
roachprod load-balancer pgurl <$cluster>
roachprod load-balancer ip <$cluster>

Get connection URL through load balancer

roachprod fetch-certs <$cluster>

Connect to cluster node via load balancer

eval ./cockroach sql --url=$(roachprod load-balancer pgurl <$cluster> --secure)

Delete all load balancers

roachprod load-balancer destroy <$cluster>

Fixes: #54176
Epic: None
Release Note: None

[cockroach-teamcity]

This change is 

[github-actions]

Potential Bug(s) Detected

The three-stage Claude Code analysis has identified potential bug(s) in this PR that may warrant investigation.

Next Steps:
Please review the detailed findings in the workflow run.

Note: When viewing the workflow output, scroll to the bottom to find the Final Analysis Summary.

After you review the findings, please tag the issue as follows:

• If the detected issue is real or was helpful in any way, please tag the issue with O-AI-Review-Real-Issue-Found
• If the detected issue was not helpful in any way, please tag the issue with O-AI-Review-Not-Helpful

[shailendra-patel]

@shailendra-patel made 2 comments.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @cpj2195 and @herkolategan).

---

pkg/roachprod/vm/aws/aws.go line 2277 at r2 (raw file):

	nlbName := loadBalancerResourceName(clusterName, port, "nlb")
	// AWS NLB names have a 32-character limit
	if len(nlbName) > 32 {

nit: Check for length on name is not required, as function loadBalancerResourceName already trim the name if length > 32.

---

pkg/roachprod/vm/aws/aws.go line 2287 at r2 (raw file):

		"--type", "network",
		"--scheme", "internet-facing",
		"--subnets",

We should consider adding a security group for NLB, this will control the inbound and outbound traffic on the NLB. I think the default security group available in zone configs should work fine.

--security-groups # aws cli flag 
// az, ok := p.Config.AZByName[v.Zone]
// az.Region.SecurityGroup this will give the security group

[shailendra-patel]

The current implementation registers each EC2 instance with the target groups. This works well for clusters where we do not scale EC2 instances up or down. However, for scale tests, we may want to consider implementing the following as part of separate PRs:

1. Support AWS Auto Scaling Groups for creating and managing instances.
2. Add support for AWS in the roachprod grow command.
3. Create AWS NLBs with Auto Scaling Groups as target groups.

Additionally, in AWS, NLBs are regional. For a multi-region cluster, you will have one NLB per region, each with a different endpoint. To simulate a regional failure, there is no single NLB endpoint that can be used as a pgurl. Therefore, we should also consider using AWS Global Accelerator on top of NLBs to support this requirement in future AWS scale tests.

This comment is not a blocker for this PR, in my opinion we need to complete the above item in order to close 153072 fully.

[cpj2195]

The current implementation registers each EC2 instance with the target groups. This works well for clusters where we do not scale EC2 instances up or down. However, for scale tests, we may want to consider implementing the following as part of separate PRs:

1. Support AWS Auto Scaling Groups for creating and managing instances.

3. Create AWS NLBs with Auto Scaling Groups as target groups.

Additionally, in AWS, NLBs are regional. For a multi-region cluster, you will have one NLB per region, each with a different endpoint. To simulate a regional failure, there is no single NLB endpoint that can be used as a pgurl. Therefore, we should also consider using AWS Global Accelerator on top of NLBs to support this requirement in future AWS scale tests.

This comment is not a blocker for this PR, in my opinion we need to complete the above item in order to close 153072 fully.

The support for addition of ASG is already part of this epic here. As for the multi region AWS cluster support, I am not able to provision a multi region roachprod cluster in aws from master as of now. I will add a new ticket to this EPIC for multi region support and try to take it up as part of that.

2. Add support for AWS in the roachprod grow command.

Will sync with you on this offline

[cpj2195]

We should consider adding a security group for NLB, this will control the inbound and outbound traffic on the NLB. I think the default security group available in zone configs should work fine.

The VM's already have security groups so any traffic will get filtered down on the VM level. Also we are not really restricting any specific traffic in the zonal config SGs so why to add an additional infrastructure component?
what do you think?

[shailendra-patel]

LGTM

[herkolategan]

Nice work! I have a few comments around ensuring we don't leak resources, and how clean-up is managed.

And then just a general question around how connections are distributed - are they round-robin?

[golgeek]

I have the same general concern @herkolategan already raised: we need to make sure that we clean up properly in case something goes wrong in the multi-step creation/deletion process. It seems like the DeleteLoadBalancer() function will properly destroy elbv2 and target-groups even if one type of resource has already been destroyed (or was never created), but we need to ensure that DeleteLoadBalancer() is properly called when deleting a cluster and when something goes wrong during LB creation.

Something else I'd like to raise: since these are new functions, you should equip them with a context.Context and pass it down the line (use ctxgroup instead of errgroup and call runJSONCommandWithContext()) as this will help in user's cancellations and operations timeout in the future.

One last thing, the pattern of "listing resources then getting elbv2 tags" is repeated multiple times. I wonder if you shouldn't move this logic to a helper function to avoid code duplication.

[golgeek]

From the awscli doc, it looks like the limit is 20 resources in a single call.
Probably not a huge near term concerns as we don't have a heavy use of LBs, but I think you should consider batching.

[cpj2195]

Parking the batching as a TODO for now since we dont use LB's currently too much.

[cpj2195]

And then just a general question around how connections are distributed - are they round-robin?

Its flow hash algorithm as per AWS docs.

[cpj2195]

bors r+

[craig]

Build succeeded:

• unit_tests
• linux_amd64_build
• local_roachtest_fips
• linux_amd64_fips_build
• local_roachtest
• docker_image_amd64
• lint
• acceptance
• check_generated_code
• examples_orms