[bug] failed to provision volume due-to already exists

[biscout42]

Describe the bug:
While creating bunch of PVCs on a single node k8s (k0s) some PVCs creation fails and stays in Pending status.
Logs from openebs-localpv-provisioner shows, that the error is

'ProvisioningFailed' failed to provision volume ... already exists

Actual reason happens because during the initial start openebs-localpv-provisioner does the PVC provision, and then fails with

request.go:690] Waited for 1.142520792s due to client-side throttling, not priority and fairness, request: GET:https://__.__.__.__:443/api/v1/namespaces/openebs/pods/init-pvc-b857cbe4-a291-4ce3-ae53-83d0a43a9a8f

leaderelection.go:330] error retrieving resource lock openebs/openebs.io-local: Get "https://__.__.__.__:443/api/v1/namespaces/openebs/endpoints/openebs.io-local": context deadline exceeded

leaderelection.go:283] failed to renew lease openebs/openebs.io-local: timed out waiting for the condition

event.go:285] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' openebs-localpv-provisioner-587f54796-pkj8v_0acbfa99-4e8b-424a-a000-57c3983d0b79 stopped leading

controller.go:889] leaderelection lost

It seems like the provisioned issued a provision request, failed to get response and then tried to provision the same PVS which leads to the error.

Expected behaviour:
Provisioner checks, that PVC exists before trying to create and fails.

Steps to reproduce the bug:
Configure a single node k8s cluster with ~20 PVCs, try to create it couple of times. You would need a bit of luck, for us it happens not very often.

The output of the following commands will help us better understand what's going on:

• kubectl get pods -n <openebs_namespace> --show-labels

kubectl get pods -n openebs --show-labels
NAME                                            READY   STATUS      RESTARTS      AGE   LABELS
init-pvc-25456a7b-0f13-4b2d-80b4-78adab349622   0/1     Completed   0             45m   <none>
init-pvc-5e8de87a-1bd5-4be6-8dcc-b2cf0f424941   0/1     Completed   0             45m   <none>
init-pvc-b857cbe4-a291-4ce3-ae53-83d0a43a9a8f   0/1     Completed   0             45m   <none>
init-pvc-c5c40720-3c36-40d2-9c70-87ae31d54a46   0/1     Completed   0             45m   <none>
openebs-localpv-provisioner-587f54796-pkj8v     1/1     Running     1 (44m ago)   45m   name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=3.2.0,pod-template-hash=587f54796
openebs-ndm-knx69                               1/1     Running     0             45m   controller-revision-hash=6db6c96bc9,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.4.0,pod-template-generation=1
openebs-ndm-operator-548d495687-8l6tv           1/1     Running     0             45m   name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=3.2.0,pod-template-hash=548d495687

• kubectl logs <upgrade_job_pod> -n <openebs_namespace>

I don't see upgrade_job_pod.

Anything else we need to know?:
Same issue was reported here openebs/openebs#3131

Environment details:

• OpenEBS version (use kubectl get po -n openebs --show-labels): see above
• Kubernetes version (use kubectl version):

Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.2+k0s

• Cloud provider or hardware configuration: single node setup with k0s, linux
• OS (e.g: cat /etc/os-release): Oracle Linux 8.10
• kernel (e.g: uname -a):
• others:

[biscout42]

Looks similar to #226