Cluster incorrectly shown as ready when both LoadBalancers are not ready

[hamzy]

/kind bug
/area provider/ibmcloud

What steps did you take and what happened:
Deploy a 4.18 cluster on a PowerVS zone where LoadBalancers are slow to create.
We are called with InfraReady. We then create DNS records for the LBs. However, only the public LB exists. So the cluster fails to deploy.

What did you expect to happen:
We should wait for all specified LBs to become ready.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[Karthik-K-N]

I think currently we check public loadbalancer and set hostname accordingly, May be we need to look for all the configured loadbalancers status before setting infra as ready.
https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/blob/main/controllers/ibmpowervscluster_controller.go#L334-L346.

[mkumatag]

@Karthik-K-N can someone fix this behaviour asap?

[Karthik-K-N]

@Karthik-K-N can someone fix this behaviour asap?

sure.

[Karthik-K-N]

I think currently we check public loadbalancer and set hostname accordingly, May be we need to look for all the configured loadbalancers status before setting infra as ready.
https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/blob/main/controllers/ibmpowervscluster_controller.go#L334-L346.

@dharaneeshvrd Just looking at the code we already check all the loadbalancer status here in ReconcileLoadbalancer https://github.com/Karthik-K-N/cluster-api-provider-ibmcloud/blob/8a563a0620f34ecb772a73bccb9ed7f1384c822f/cloud/scope/powervs_cluster.go#L1977-L1979,

when do you think this issue will occur.

[Karthik-K-N]

We did some investigation on the code and its working as expected

1. For all the LB configured for the cluster, We requue if the LB status is not active, For reference code.
2. We set infra.Ready only after LB are ready, For reference code.

@hamzy are you running with latest code from this repo?
If you still face issues with latest code, Please help us with controller logs for further debugging. Thanks

[Karthik-K-N]

/triage needs-information

[hamzy]

We are running version
sigs.k8s.io/cluster-api-provider-ibmcloud v0.9.0-alpha.0.0.20240913094112-c6bcd313bce0

https://github.com/openshift/installer/blob/master/cluster-api/providers/ibmcloud/go.mod#L7

[hamzy]

Version sigs.k8s.io/cluster-api-provider-ibmcloud v0.9.0-beta.0.0.20241017140904-8a563a0620f3
in openshift/installer#9118
also fails.

[Karthik-K-N]

Could you please share the error and the code reference.

[hamzy]

This is a failing run of #9118
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_installer/9118/pull-ci-openshift-installer-master-e2e-powervs-capi-ovn/1849180644286926848/artifacts/e2e-powervs-capi-ovn/ipi-install-powervs-install/artifacts/clusterapi_output/

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_installer/9118/pull-ci-openshift-installer-master-e2e-powervs-capi-ovn/1849180644286926848/artifacts/e2e-powervs-capi-ovn/ipi-install-powervs-install/artifacts/.openshift_install.log

Search for "InfraReady: hostname =" to see there is only one LB active at the time although the other one does become active eventually.