Deployment failure on Azure Kubernetes: "FailedAttachVolume"

[OmpahDev]

When trying to deploy rabbitmq using the following StatefulSet to Azure AKS v1.26.3:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nxd-rabbitmq
  namespace: "default"
  labels:
    app.kubernetes.io/name: rabbitmq
    helm.sh/chart: rabbitmq-12.0.2
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
spec:
  serviceName: nxd-rabbitmq-headless
  podManagementPolicy: OrderedReady
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: rabbitmq
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels:
        app.kubernetes.io/name: rabbitmq
        helm.sh/chart: rabbitmq-12.0.2
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/managed-by: Helm
      annotations:
        checksum/config: <redacted>
        checksum/secret: <redacted>
    spec:

      serviceAccountName: nxd-rabbitmq
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: rabbitmq
                    app.kubernetes.io/instance: release-name
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      securityContext:
        fsGroup: 1001
      terminationGracePeriodSeconds: 120
      initContainers:
      containers:
        - name: rabbitmq
          image: docker.io/bitnami/rabbitmq:3.12.0-debian-11-r0
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/bash
                  - -ec
                  - |
                    if [[ -f /opt/bitnami/scripts/rabbitmq/nodeshutdown.sh ]]; then
                        /opt/bitnami/scripts/rabbitmq/nodeshutdown.sh -t "120" -d "false"
                    else
                        rabbitmqctl stop_app
                    fi
          env:
            - name: BITNAMI_DEBUG
              value: "false"
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: K8S_SERVICE_NAME
              value: nxd-rabbitmq-headless
            - name: K8S_ADDRESS_TYPE
              value: hostname
            - name: RABBITMQ_FORCE_BOOT
              value: "no"
            - name: RABBITMQ_NODE_NAME
              value: "rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local"
            - name: K8S_HOSTNAME_SUFFIX
              value: ".$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local"
            - name: RABBITMQ_MNESIA_DIR
              value: "/bitnami/rabbitmq/mnesia/$(RABBITMQ_NODE_NAME)"
            - name: RABBITMQ_LDAP_ENABLE
              value: "no"
            - name: RABBITMQ_LOGS
              value: "-"
            - name: RABBITMQ_ULIMIT_NOFILES
              value: "65536"
            - name: RABBITMQ_USE_LONGNAME
              value: "true"
            - name: RABBITMQ_ERL_COOKIE
              valueFrom:
                secretKeyRef:
                  name: nxd-rabbitmq
                  key: rabbitmq-erlang-cookie
            - name: RABBITMQ_LOAD_DEFINITIONS
              value: "no"
            - name: RABBITMQ_DEFINITIONS_FILE
              value: "/app/load_definition.json"
            - name: RABBITMQ_SECURE_PASSWORD
              value: "yes"
            - name: RABBITMQ_USERNAME
              value: "guest"
            - name: RABBITMQ_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: nxd-rabbitmq
                  key: rabbitmq-password
            - name: RABBITMQ_PLUGINS
              value: "rabbitmq_management, rabbitmq_peer_discovery_k8s, rabbitmq_auth_backend_ldap"
          envFrom:
          ports:
            - name: amqp
              containerPort: 5672
            - name: dist
              containerPort: 25672
            - name: stats
              containerPort: 15672
            - name: epmd
              containerPort: 4369
          livenessProbe:
            failureThreshold: 6
            initialDelaySeconds: 120
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 20
            exec:
              command:
                - sh
                - -ec
                - curl -f --user guest:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 20
            exec:
              command:
                - sh
                - -ec
                - curl -f --user guest:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms
          resources:
            limits:
              memory: 256Mi
            requests:
              memory: 256Mi
          volumeMounts:
            - name: configuration
              mountPath: /bitnami/rabbitmq/conf
            - name: data
              mountPath: /bitnami/rabbitmq/mnesia
      volumes:
        - name: configuration
          projected:
            sources:
              - secret:
                  name: nxd-rabbitmq-config
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: data
        labels:
          app.kubernetes.io/name: rabbitmq
          app.kubernetes.io/instance: release-name
      spec:
        accessModes:
            - "ReadWriteOnce"
        resources:
          requests:
            storage: "8Gi"
---

The pod keeps failing to start because of volume attachment issues:

FailedAttachVolume AttachVolume.Attach failed for volume "pvc-a253bbf3-bb65-4730-8d7c-a88378dfafcc" : rpc error: code = Internal desc = Attach volume /subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Compute/disks/pvc-a253bbf3-bb65-4730-8d7c-a88378dfafcc to instance aks-agentpool-40990672-vmss000001 failed with could not find disk(/subscriptions/<redacted>/resourcegroups/<redacted>/providers/microsoft.compute/disks/pvc-a253bbf3-bb65-4730-8d7c-a88378dfafcc) in current disk list(len: 0) nor in diskMap(map[])

FailedMount Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-czrlf configuration data]: timed out waiting for the condition

[lukebakken]

This discussion is probably best posted here: https://github.com/bitnami/charts

[OmpahDev]

Tried that - they pointed me back here, saying it's not issue with bitnami but with rabbitmq

[lukebakken]

Do you have a link with the justification from that project? As you can see, this repo really doesn't have anything to do with kubernetes.

Also, please see @michaelklishin's comment.

[OmpahDev]

bitnami/charts#17716 (comment)

[mkuratczyk]

I'd say this is an infrastructure-level issue. RabbitMQ isn't aware of PVCs. Your StatefulSet definition is, but as long as the definition is correct, it's all in the hands of the Kubernetes cluster and underlying layers to make it work.

[michaelklishin]

I doubt that

FailedMount           Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-czrlf configuration data]: timed out waiting for the condition

is an issue with RabbitMQ itself. Looks like double mounting a volume would produce exactly the same error.

[OmpahDev]

I'm not double mounting, already checked for that