vpc e2e prow job is failing

[mkumatag]

/kind bug
/area provider/ibmcloud

What steps did you take and what happened:
https://prow.ppc64le-cloud.org/job-history/gs/ppc64le-kubernetes/logs/periodic-capi-provider-ibmcoud-e2e-vpc

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[Prajyot-Parab]

/assign @Prajyot-Parab

[Prajyot-Parab]

Few findings post VPC E2E debugging

• The nil reference error is as a result of previous e2e run infra being left behind
• Need to ensure cleanup is done properly even in case of failure (I see LB, Subnet, VPC) existing from previous runs
• can we append test vpc name with some random string instead of current pattern as I think it conflicts with pervious run VPC if left behind

[Prajyot-Parab]

Update -
The VPC CI is failing as the ClusterResourceSet resources are not getting applied, which results in dependent resources not getting applied on VPC cluster. As a result the providerID is not being set at the Node level hence the ginkgo e2e flow assumes the ControlPlane machine not to exists and e2e fails.

We manually applied the ClusterResourceSet based dependent resources on ControlPlane directly and providerID was set post that at Node level.

Error seen with ClusterResourceSet -

Status:
  Conditions:
    Last Transition Time:  2023-02-24T09:39:29Z
    Message:               failed to create cluster accessor: error creating dynamic rest mapper for remote cluster "create-workload-cluster-f5g8d4/capi-ibmcloud-e2e-foonnz": Get "https://ab92ddd6-ca-tor.lb.appdomain.cloud:6443/api?timeout=10s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    Reason:                RemoteClusterClientFailed
    Severity:              Error
    Status:                False
    Type:                  ResourcesApplied

On further debugging we were able to verify that LoadBalancer Address is reachable from Pod inside Prow where the actual run happens.
Tried to delete and reapply the ClusterResourseSet but same error was observed.

/cc @Amulyam24 Please add any other info if missing
/cc @mkumatag