🐛 Reconcile target groups and listeners as their own entities #5004

nrb · 2024-06-05T21:17:48Z

Signed-off-by: Nolan Brubaker [email protected]

What type of PR is this?

/kind bug

What this PR does / why we need it:

If a load balancer is created but the calls to create a target group or
listener fails (for example, due to a rate limit), creating the group or
listener is never retried. This results in a load balancer that's
created, but there is nothing attached to it, making it useless.

This change reconciles the target groups and listeners as their own entities, so that even if they fail to create initially, we'll keep retrying.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5002

Special notes for your reviewer:

Checklist:

Release note:

    Reconcile target groups and listeners independently of load balancers to handle failures more gracefully.

If a load balancer is created but the calls to create a target group or listener fails (for example, due to a rate limit), creating the group or listener is never retried. This results in a load balancer that's created, but there is nothing attached to it, making it useless. Signed-off-by: Nolan Brubaker <[email protected]>

k8s-ci-robot · 2024-06-05T21:17:51Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

pkg/cloud/services/elb/loadbalancer.go

Signed-off-by: Nolan Brubaker <[email protected]>

nrb · 2024-06-06T18:45:03Z

/test ?

k8s-ci-robot · 2024-06-06T18:45:05Z

@nrb: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-aws-build
/test pull-cluster-api-provider-aws-build-docker
/test pull-cluster-api-provider-aws-test
/test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-aws-apidiff-main
/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass
/test pull-cluster-api-provider-aws-e2e-conformance
/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
/test pull-cluster-api-provider-aws-e2e-eks
/test pull-cluster-api-provider-aws-e2e-eks-gc
/test pull-cluster-api-provider-aws-e2e-eks-testing

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-aws-apidiff-main
pull-cluster-api-provider-aws-build
pull-cluster-api-provider-aws-build-docker
pull-cluster-api-provider-aws-test
pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nrb · 2024-06-06T18:45:38Z

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass

nrb · 2024-06-06T18:48:43Z

/retitle 🐛 Reconcile target groups and listeners as their own entities

nrb · 2024-06-06T18:48:50Z

/assign @damdo

Signed-off-by: Nolan Brubaker <[email protected]>

nrb · 2024-06-06T19:34:10Z

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass

pkg/cloud/services/elb/loadbalancer.go

* reconcileTargetGroupsAndListeners needs both the desired specification as well as the ARN assigned by the AWS API calls. * Rename `spec` in reconcileV2LB to `desiredLB` to clarify what data is actually held in it. Signed-off-by: Nolan Brubaker <[email protected]>

nrb · 2024-06-07T16:51:12Z

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass

@r4f4 I clarified some of the variables and updated the arguments for reconcileTargetGroupsAndListeners. 🤞 that e2es are happy again.

r4f4 · 2024-06-07T17:06:07Z

LGTM. I'll give it a try again.

r4f4 · 2024-06-07T18:17:47Z

Got a successful install this time.
/lgtm

k8s-ci-robot · 2024-06-07T18:17:51Z

@r4f4: changing LGTM is restricted to collaborators

In response to this:

Got a successful install this time.
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

damdo

Mostly LGTM, just a question/comment.

damdo · 2024-06-10T13:26:36Z

pkg/cloud/services/elb/loadbalancer.go

+	if len(existingTargetGroups.TargetGroups) == len(existingListeners.Listeners) && len(existingListeners.Listeners) == len(spec.ELBListeners) {
+		return existingTargetGroups.TargetGroups, existingListeners.Listeners, nil
+	}


Could we add a comment on why this is necessary? Is this an optimization?

Also is this condition always a guarantee there is nothing to do?
Especially looking at, could there be a case where the count is off and doing g.TargetGroupName == ln.TargetGroup.Name naming matching would be better?

len(existingTargetGroups.TargetGroups) == len(existingListeners.Listeners)

Yeah, it was an attempt to optimize so we don't go through the full loop every time.

Matching on the name might be better. I suppose it's possible that a user could create more target groups in AWS than CAPA knows about, though I'm not sure why they'd do that.

In any case, if we check that the target group names line up between AWS and our spec, we're still repeating the loop every time. The main benefit there would be that we could potentially skip doing more API calls within a single iteration.

One reason we saw the LB creation fail was because of API call rate limiting in very busy envs, so avoiding extra calls when possible would be nice.

Thinking this through some more, I think the only numeric comparison we can do that's valid all the time is if len(existingTargetGroups) < len(desiredTargetGroups) { reconcile }. Even if the lists are equal in length, it doesn't guarantee that they're the exact same names.

I'll look through this and see if I can do any further optimizations, but I'll at least remove this length check.

Thanks for the productive back-and-forth here.

Please consider a future PR for optimizations, because this PR is already complex (although the new tests are very helpful, they themselves are complex). 🙏

dlipovetsky · 2024-06-10T16:59:12Z

/milestone v2.5.1

dlipovetsky · 2024-06-10T17:19:22Z

pkg/cloud/services/elb/loadbalancer_test.go

+		spec          func(spec infrav1.LoadBalancer) infrav1.LoadBalancer
+	}{
+		{
+			name: "main create flow",


Consider making this more descriptive

dlipovetsky · 2024-06-10T17:21:54Z

/approve

k8s-ci-robot · 2024-06-10T17:22:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dlipovetsky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dlipovetsky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dlipovetsky · 2024-06-10T17:25:22Z

/milestone v2.6.0
/milestone v2.5.1

When the optimization was enabled, we were missing a bunch of creation calls that need to get defined. Add some partial matcher support in order to allow for generated names. Signed-off-by: Nolan Brubaker <[email protected]>

damdo

/lgtm

/hold
@nrb feel free to unhold once happy with it merging

nrb · 2024-06-11T15:13:35Z

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass

nrb · 2024-06-11T23:45:27Z

/unhold

k8s-ci-robot requested review from fiunchinho and vincepri June 5, 2024 21:17

nrb commented Jun 5, 2024

View reviewed changes

pkg/cloud/services/elb/loadbalancer.go Outdated Show resolved Hide resolved

nrb commented Jun 5, 2024

View reviewed changes

pkg/cloud/services/elb/loadbalancer.go Outdated Show resolved Hide resolved

nrb added 2 commits June 6, 2024 13:46

Look up existing target groups and listeners

0991311

Signed-off-by: Nolan Brubaker <[email protected]>

Update unit tests for target groups and listeners

158fc75

Signed-off-by: Nolan Brubaker <[email protected]>

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 6, 2024

nrb marked this pull request as ready for review June 6, 2024 18:48

k8s-ci-robot requested a review from dlipovetsky June 6, 2024 18:48

k8s-ci-robot changed the title ~~WIP: 🐛 Reconcile target groups and listeners as their own entities~~ 🐛 Reconcile target groups and listeners as their own entities Jun 6, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 6, 2024

k8s-ci-robot assigned damdo Jun 6, 2024

Fix linting issues

a49371d

Signed-off-by: Nolan Brubaker <[email protected]>

nrb commented Jun 7, 2024

View reviewed changes

pkg/cloud/services/elb/loadbalancer.go Outdated Show resolved Hide resolved

r4f4 reviewed Jun 7, 2024

View reviewed changes

pkg/cloud/services/elb/loadbalancer.go Outdated Show resolved Hide resolved

damdo reviewed Jun 10, 2024

View reviewed changes

k8s-ci-robot added this to the v2.5.1 milestone Jun 10, 2024

dlipovetsky reviewed Jun 10, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 10, 2024

k8s-ci-robot modified the milestones: v2.5.1, v2.6.0 Jun 10, 2024

Remove incorrect optimization and fixup tests

cc2cfe4

When the optimization was enabled, we were missing a bunch of creation calls that need to get defined. Add some partial matcher support in order to allow for generated names. Signed-off-by: Nolan Brubaker <[email protected]>

damdo reviewed Jun 11, 2024

View reviewed changes

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Jun 11, 2024

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 11, 2024

k8s-ci-robot merged commit daedbe0 into kubernetes-sigs:main Jun 11, 2024

This was referenced Jun 12, 2024

Duplicate target groups and listeners created in the ELBv2 reconcile loop #5015

Closed

OCPBUGS-34819: capi/aws: bump provider for fixes openshift/installer#8545

Merged

🐛 Reconcile target groups and listeners as their own entities #5004

🐛 Reconcile target groups and listeners as their own entities #5004

Uh oh!

Conversation

nrb commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jun 5, 2024

Uh oh!

Uh oh!

Uh oh!

nrb commented Jun 6, 2024

Uh oh!

k8s-ci-robot commented Jun 6, 2024

Uh oh!

nrb commented Jun 6, 2024

Uh oh!

nrb commented Jun 6, 2024

Uh oh!

nrb commented Jun 6, 2024

Uh oh!

nrb commented Jun 6, 2024

Uh oh!

Uh oh!

Uh oh!

nrb commented Jun 7, 2024

Uh oh!

r4f4 commented Jun 7, 2024

Uh oh!

r4f4 commented Jun 7, 2024

Uh oh!

k8s-ci-robot commented Jun 7, 2024

Uh oh!

damdo left a comment

Choose a reason for hiding this comment

Uh oh!

damdo Jun 10, 2024

Choose a reason for hiding this comment

Uh oh!

nrb Jun 10, 2024

Choose a reason for hiding this comment

Uh oh!

r4f4 Jun 10, 2024

Choose a reason for hiding this comment

Uh oh!

nrb Jun 10, 2024

Choose a reason for hiding this comment

Uh oh!

dlipovetsky Jun 10, 2024

Choose a reason for hiding this comment

Uh oh!

dlipovetsky commented Jun 10, 2024

Uh oh!

dlipovetsky Jun 10, 2024

Choose a reason for hiding this comment

Uh oh!

dlipovetsky commented Jun 10, 2024

Uh oh!

k8s-ci-robot commented Jun 10, 2024

Uh oh!

dlipovetsky commented Jun 10, 2024

Uh oh!

damdo left a comment

Choose a reason for hiding this comment

Uh oh!

nrb commented Jun 11, 2024

Uh oh!

nrb commented Jun 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nrb commented Jun 5, 2024 •

edited

Loading