Allow deleting LBs in ERROR status (kubernetes#2037)

In practice Octavia is quite an unreliable system. We should expect that
load balancers will occasionally go into ERROR state for reasons as
elaborate as race conditions and as simple as lack of resources to spawn
a new Amphora VM.

This commit makes sure that we allow deleting LBs in ERROR state, so
that Services served by such LBs could be deleted and their creation
retried.

Obviously there's room for improvement, as we should probably detect LBs
going into ERROR and attempt to delete and recreate them ourselves, but
this patch is a low hanging fruit fix for the issue.

Author: dulek

pkg/openstack/loadbalancer.go

@@ -64,6 +64,7 @@ const (
 	defaultLoadBalancerSourceRangesIPv4 = "0.0.0.0/0"
 	defaultLoadBalancerSourceRangesIPv6 = "::/0"
 	activeStatus                        = "ACTIVE"
+	errorStatus                         = "ERROR"
 	annotationXForwardedFor             = "X-Forwarded-For"
 
 	ServiceAnnotationLoadBalancerInternal             = "service.beta.kubernetes.io/openstack-internal-load-balancer"
@@ -3462,8 +3463,8 @@ func (lbaas *LbaasV2) ensureLoadBalancerDeleted(ctx context.Context, clusterName
 		return nil
 	}
 
-	if loadbalancer.ProvisioningStatus != activeStatus {
-		return fmt.Errorf("load balancer %s is not ACTIVE, current provisioning status: %s", loadbalancer.ID, loadbalancer.ProvisioningStatus)
+	if loadbalancer.ProvisioningStatus != activeStatus && loadbalancer.ProvisioningStatus != errorStatus {
+		return fmt.Errorf("load balancer %s is in immutable status, current provisioning status: %s", loadbalancer.ID, loadbalancer.ProvisioningStatus)
 	}
 
 	if strings.HasPrefix(loadbalancer.Name, servicePrefix) {