How to run a RabbitmqCluster using Rabbitmq operator and csi-hostpath-driver?

[jalvespinto]

I am unsuccessfully trying to test bitnami rabbitmq-cluster-operator using minikube with "csi-hostpath-driver" addon. The reason I am using the addon is because the default host-path volume provisioner doesn’t support multi-node clusters. My guess is that I have to include some parameters/annotations to the csi-hostpath-sc storageclass, but I am not sure and don't know what it would be.

minikube version:
W0331 13:44:08.650859 295823 main.go:291] Unable to resolve the current Docker CLI context "default": context "default" does not exist
minikube version: v1.29.0
commit: ddac20b4b34a9c8c857fc602203b6ba2679794d3

docker --version:
Docker version 23.0.2, build 569dd73

kubectl version --short:
Client Version: v1.26.3
Kustomize Version: v4.5.7
Server Version: v1.25.8

minikube start args (for now without multi-nodes):
"--kubernetes-version=v1.25.8",
"--driver=kvm2",
"--cpus=6",
"--memory=8064",
"--disk-size=50g",
"--addons=ingress",
"--addons=volumesnapshots",
"--addons=csi-hostpath-driver"

The addon automatically creates a storageClass named csi-hostpath-sc that I am trying to use as the StorageClassName on the RabbitmqCluster:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-hostpath-sc  
  uid: ecd3406c-26db-43d4-b1a7-ed7ded88b5d9
  resourceVersion: '375'
  creationTimestamp: '2023-03-31T15:00:36Z'
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile"},"name":"csi-hostpath-sc"},"provisioner":"hostpath.csi.k8s.io","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
  managedFields:
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: storage.k8s.io/v1
      time: '2023-03-31T15:00:36Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:addonmanager.kubernetes.io/mode: {}
        f:provisioner: {}
        f:reclaimPolicy: {}
        f:volumeBindingMode: {}
  selfLink: /apis/storage.k8s.io/v1/storageclasses/csi-hostpath-sc
allowedTopologies: []
mountOptions: []
parameters: {}
provisioner: hostpath.csi.k8s.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

The operator runs fine. Now, for the RabbitmqCluster manifest, if I don't declare a persistence.storageClassName it will use the default hostpath provisioner and will run successfully. But if I use the csi-hostpath-driver storageclass it wont't.

kubectl describe pv pvc-8828a7ff-fe32-4052-8edd-edb717e6f6dc -n rabbitmq:

Name:              pvc-8828a7ff-fe32-4052-8edd-edb717e6f6dc
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: hostpath.csi.k8s.io
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      csi-hostpath-sc
Status:            Bound
Claim:             rabbitmq/persistence-rabbitmq-cluster-server-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          10Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.hostpath.csi/node in [minikube]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            hostpath.csi.k8s.io
    FSType:            
    VolumeHandle:      fb104259-cfe3-11ed-9431-9ab49114a143
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1680274959332-8081-hostpath.csi.k8s.io
Events:                <none>

kubectl describe pvc persistence-rabbitmq-cluster-server-0 -n rabbitmq:

Name:          persistence-rabbitmq-cluster-server-0
Namespace:     rabbitmq
StorageClass:  csi-hostpath-sc
Status:        Bound
Volume:        pvc-8828a7ff-fe32-4052-8edd-edb717e6f6dc
Labels:        app.kubernetes.io/component=rabbitmq
               app.kubernetes.io/name=rabbitmq-cluster
               app.kubernetes.io/part-of=rabbitmq
Annotations:   meta.helm.sh/release-name: rabbitmq-cluster
               meta.helm.sh/release-namespace: rabbitmq
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: hostpath.csi.k8s.io
               volume.kubernetes.io/storage-provisioner: hostpath.csi.k8s.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       rabbitmq-cluster-server-0
Events:
  Type     Reason                   Age                    From                                                                                 Message
  ----     ------                   ----                   ----                                                                                 -------
  Normal   ExternalProvisioning     3m21s (x2 over 3m21s)  persistentvolume-controller                                                          waiting for a volume to be created, either by external provisioner "hostpath.csi.k8s.io" or manually created by system administrator
  Normal   Provisioning             3m21s                  hostpath.csi.k8s.io_csi-hostpath-provisioner-0_c7aa6927-8f3b-4adc-a41f-91ec72ebf9be  External provisioner is provisioning volume for claim "rabbitmq/persistence-rabbitmq-cluster-server-0"
  Normal   ProvisioningSucceeded    3m21s                  hostpath.csi.k8s.io_csi-hostpath-provisioner-0_c7aa6927-8f3b-4adc-a41f-91ec72ebf9be  Successfully provisioned volume pvc-8828a7ff-fe32-4052-8edd-edb717e6f6dc
  Warning  VolumeConditionAbnormal  1s                     csi-pv-monitor-controller-hostpath.csi.k8s.io                                        The capacity of volume is greater than actual storage

Here I have no idea why it says the capacity of volume is greater than actual storage.
kubectl describe node minikube:

Capacity:
  cpu:                6
  ephemeral-storage:  45604772Ki
  hugepages-2Mi:      0
  memory:             8005560Ki
  pods:               110
Allocatable:
  cpu:                6
  ephemeral-storage:  45604772Ki
  hugepages-2Mi:      0
  memory:             8005560Ki
  pods:               110
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                1150m (19%)  500m (8%)
  memory             760Mi (9%)   1194Mi (15%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:              <none>

Finally, these are the events for kubectl describe po rabbitmq-cluster-server-0 -n rabbitmq:

Events:
  Type     Reason                   Age                   From                                      Message
  ----     ------                   ----                  ----                                      -------
  Warning  FailedScheduling         11m                   default-scheduler                         0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled                11m                   default-scheduler                         Successfully assigned rabbitmq/rabbitmq-cluster-server-0 to minikube
  Warning  VolumeConditionAbnormal  11m (x10 over 11m)    csi-pv-monitor-agent-hostpath.csi.k8s.io  The volume isn't mounted
  Normal   SuccessfulAttachVolume   11m                   attachdetach-controller                   AttachVolume.Attach succeeded for volume "pvc-8828a7ff-fe32-4052-8edd-edb717e6f6dc"
  Normal   Pulled                   10m                   kubelet                                   Container image "docker.io/bitnami/rabbitmq:3.10.19-debian-11-r4" already present on machine
  Normal   Created                  10m                   kubelet                                   Created container setup-container
  Normal   Started                  10m                   kubelet                                   Started container setup-container
  Normal   Pulled                   8m23s (x2 over 10m)   kubelet                                   Container image "docker.io/bitnami/rabbitmq:3.10.19-debian-11-r4" already present on machine
  Normal   Created                  8m23s (x2 over 10m)   kubelet                                   Created container rabbitmq
  Normal   Started                  8m23s (x2 over 10m)   kubelet                                   Started container rabbitmq
  Warning  Unhealthy                5m49s (x26 over 10m)  kubelet                                   Readiness probe failed: dial tcp 10.244.0.19:5672: connect: connection refused
  Normal   VolumeConditionNormal    62s (x91 over 10m)    csi-pv-monitor-agent-hostpath.csi.k8s.io  The Volume returns to the healthy state
  Warning  BackOff                  59s (x7 over 6m20s)   kubelet                                   Back-off restarting failed container