🐛 Improve handling of missing load balancer permissions

[sd109]

What this PR does / why we need it:

Currently, when a user tries to create a cluster using OpenStack credentials which are missing the load balancer permissions, CAPO adds the finalized to the OpenStackCluster resource then fails to create the load balancer. When the user then tries to delete the cluster, CAPO makes a GET request to the Octavia API to get the load balancer details and receives a 403 (permission denied) response, so the only way to allow the cluster deletion to proceed is to manually remove the finalizer from the OpenStackCluster resource.

This change prevents the above edge case by only attempting to delete the API server load balancer if the load balancer ID is populated in the OpenStackCluster's status field.

TODOs:

• squashed commits
• if necessary:
  • includes documentation
  • adds unit tests

/hold

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: sd109 / name: Scott Davidson (35cd30d, 22820cb)

[netlify]

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

Name	Link
🔨 Latest commit	22820cb
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-cluster-api-openstack/deploys/689b4b8d8cadd30009e8b927
😎 Deploy Preview	https://deploy-preview-2629--kubernetes-sigs-cluster-api-openstack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Welcome @sd109!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-openstack 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-openstack has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @sd109. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[bnallapeta]

@sd109

Thank you for this PR. The change you've proposed to check the status field certainly fixes the immediate issue. However, there's a more robust way to solve this that avoids a potential race condition (where the LB is created but the status isn't updated).

Instead of avoiding the call to delete the load balancer, I think it's better to make the DeleteLoadBalancer function itself more resilient. Could you modify the code so that when DeleteLoadBalancer receives a 403 Forbidden error from the OpenStack API, it logs a warning and returns nil?

This would handle the permission error gracefully, allowing cluster deletion to proceed without relying on the status field being in sync. Once that change is made, adding a unit test for that specific error handling logic would be the final step.

Let me know what you think of this approach.

[EmilienM]

Right now, CAPO only supports Octavia when a load balancer is being enabled via openStackCluster.Spec.APIServerLoadBalancer.IsEnabled.
If your cloud doesn't have Octavia or you don't want Octavia to handle the API Server load balancing, you shouldn't set this to True because CAPO doesn't support that now.

Matt had in mind a major rework for our load balancer support, having some sort of a way to define a load balancer provider (octavia, metallb, etc) but this hasn't been done.

The reason why I don't want this patch is that you should not have an failure to delete a load balancer if openStackCluster.Spec.APIServerLoadBalancer.IsEnabled was initially set to enabled; simply because if you could have created it, you should be able to disable it.

Let me know if I missed something, I'm happy to discuss further and refine here a fix that works for you.

[bnallapeta]

@EmilienM

While I agree users shouldn't enable LB without proper permissions, OpenStack permission models can be complex (e.g., IAM policies that allow create but not delete, or permissions that change between cluster lifecycle events). A more robust solution would be to make DeleteLoadBalancer handle 403 errors gracefully rather than avoiding the call entirely - this way we still attempt deletion as you suggest, but don't leave users stuck with unremovable finalizers when permissions are insufficient.

[sd109]

Right now, CAPO only supports Octavia when a load balancer is being enabled via openStackCluster.Spec.APIServerLoadBalancer.IsEnabled. If your cloud doesn't have Octavia or you don't want Octavia to handle the API Server load balancing, you shouldn't set this to True because CAPO doesn't support that now.

@EmilienM agreed, but even with Octavia-enabled clouds we still see problems arise when other tools are used to wrap CAPO functionality, since users are not always explicitly aware that APIServerLoadBalancer.IsEnabled is being set for them.

For example, when using the magnum-capi-helm driver a user might execute an openstack coe cluster create command using an app cred which is missing the load-balancer_member role. In such a situation, the cluster creation is stuck in an indefinite CREATING state because of the permission denied on the LB create API call and then when a user later tries to delete the Magnum cluster it instead gets stuck in a DELETING state indefinitely until the cloud operator manually removes the relevant finaliser from the OpenStackCluster object on the management cluster (which the user cannot do themselves because the CAPI management cluster is a cloud admin responsibility).

We originally proposed a fix on the Magnum driver side here, perhaps the previous discussion over there will help to provide some additional context.

[lentzi90]

/ok-to-test