Multi-Cluster Ceph issue with CephFS

[seanmcconkey]

Hi, rather than jump in and raise an Issue, I thought I'd start here. I've read all the documentation and read threads in here, however I cannot get multi-cluster to work.

Firstly my approach is slightly different, in that I am not using CephCSI to provision volumes. These are created externally on the Ceph Cluster, so I'm using a Static volume approach.

The error we're seeing from the ceph-csi-cephfs-provisioner is

I0217 16:33:22.379849       1 utils.go:350] ID: 1484 Req-ID: algol-pv GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0217 16:33:22.379893       1 utils.go:351] ID: 1484 Req-ID: algol-pv GRPC request: {"node_id":"dev-workers-t7tbd-t2n4n","volume_id":"algol-pv"}
W0217 16:33:22.380126       1 controllerserver.go:1169] ID: 1484 Req-ID: algol-pv controller publish secret not found: failed to decode volume ID (algol-pv): failed to decode CSI identifier, string underflow

You'd expect this to be a misconfigured Storage Class because it says it can't find the controller publish secret although its not, see below.

I've checked the RBAC, RolesBindings, Roles etc against the examples in this repo, all good except for the nodeplugin role, which isn't created for some reason.

Apart from that I can find no other issues, the error we see in the pod description is at the bottom of this post.

Current deployment configs and info.

- K8s Version - v.1.34.3
- ceph-csi Helm Release - 3.16.1

Configmaps - note hostnames are slightly redacted and are in use on other single config deployments without issues, I have also created the storage classes & secrets in the csi-ceph-system namespace but get the same results.

config.json

              csiConfig:
                - "clusterID": "2660d5ed-e7ef-4950-a051-c84a0b4e78b0"
                  "monitors":
                    - "deneb-dev-mon1"
                    - "deneb-dev-mon2"
                    - "deneb-dev-mon3"
    
                - "clusterID": "68b8c702-e575-11ef-91e4-109819979c9d"
                  "monitors":
                    - "data-ceph-algol-mon1"
                    - "data-ceph-algol-mon2"
                    - "data-ceph-algol-mon3"

apiVersion: v1
data:
  ceph.conf: |
    [global]
      auth supported = cephx
      auth_cluster_required = cephx
      auth_service_required = cephx
      auth_client_required = cephx
  keyring: ""
kind: ConfigMap

Storage Classes

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-cephfs-deneb-sc
provisioner: cephfs.csi.ceph.com
parameters:
   clusterID: 2660d5ed-e7ef-4950-a051-c84a0b4e78b0
   fsName: cephfs
   csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-deneb-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-test
   csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-deneb-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-test
   csi.storage.k8s.io/controller-publish-secret-name: csi-cephfs-deneb-secret
   csi.storage.k8s.io/controller-publish-secret-namespace: ceph-test
   csi.storage.k8s.io/node-publish-secret-name: csi-cephfs-deneb-secret
   csi.storage.k8s.io/node-publish-secret-namespace: ceph-test
   csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-deneb-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-test
   csi.storage.k8s.io/snapshotter-secret-name: csi-cephfs-deneb-secret
   csi.storage.k8s.io/snapshotter-secret-namespace: ceph-test
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions: []

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-cephfs-algol-sc
provisioner: cephfs.csi.ceph.com
parameters:
   clusterID: 68b8c702-e575-11ef-91e4-109819979c9d
   fsName: cephfs
   csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-algol-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-test
   csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-algol-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-test
   csi.storage.k8s.io/controller-publish-secret-name: csi-cephfs-algol-secret
   csi.storage.k8s.io/controller-publish-secret-namespace: ceph-test
   csi.storage.k8s.io/node-publish-secret-name: csi-cephfs-algol-secret
   csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-algol-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-test
   csi.storage.k8s.io/snapshotter-secret-name: csi-cephfs-algol-secret
   csi.storage.k8s.io/snapshotter-secret-namespace: ceph-test
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions: []

PV

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: deneb-pv
spec:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 1Pi
  storageClassName: csi-cephfs-deneb-sc
  csi:
    driver: cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: csi-cephfs-deneb-secret
      namespace: ceph-test
    volumeAttributes:
      "clusterID": "2660d5ed-e7ef-4950-a051-c84a0b4e78b0"
      "staticVolume": "true"
      "rootPath": "/ska"
      "fsName": "cephfs"
    volumeHandle: deneb-pv
  persistentVolumeReclaimPolicy: Delete
  volumeMode: Filesystem

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: algol-pv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 100Ti
  storageClassName: csi-cephfs-algol-sc
  csi:
    driver: cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: csi-cephfs-algol-secret
      namespace: ceph-test
    volumeAttributes:
      "clusterID": "68b8c702-e575-11ef-91e4-109819979c9d"
      "staticVolume": "true"
      "rootPath": "/data"
      "fsName": "cephfs"
    volumeHandle: algol-pv
  persistentVolumeReclaimPolicy: Delete
  volumeMode: Filesystem

PVCs

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: deneb-pvc
  namespace: ceph-test
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 1Pi
  storageClassName: "csi-cephfs-deneb-sc"
  volumeMode: Filesystem
  volumeName: deneb-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: algol-pvc
  namespace: ceph-test
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100Ti
  storageClassName: "csi-cephfs-algol-sc"
  volumeMode: Filesystem
  volumeName: algol-pv

Secrets

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-deneb-secret
  namespace: ceph-test
type: Opaque
stringData:
  userID: ro-user
  userKey: xxxxx

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-algol-secret
  namespace: ceph-test
type: Opaque
stringData:
  userID: users-rw
  userKey: xxxxx

A test pod.

apiVersion: v1
kind: Pod
metadata:
  name: ceph-clusters
  namespace: ceph-test
spec:
  volumes:
    - name: deneb-pv
      persistentVolumeClaim:
        claimName: deneb-pvc
        readOnly: true
    - name: algol-pv
      persistentVolumeClaim:
        claimName: algol-pvc
        readOnly: false
  containers:
    - name: ceph-clusters
      image: busybox
      command: ["/bin/sleep", "1h"]
      volumeMounts:
        - name: deneb-pv
          mountPath: /deneb
        - name: algol-pv
          mountPath: /algol

Pod errors.

  Normal   Scheduled               23m                  default-scheduler        Successfully assigned ceph-test/ceph-clusters to uksrc-ral-dev-workers-t7tbd-t2n4n
  Normal   SuccessfulAttachVolume  23m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "algol-pv"
  Normal   SuccessfulAttachVolume  23m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "deneb-pv"
  Warning  FailedMount             23m                  kubelet                  MountVolume.MountDevice failed for volume "deneb-pv" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph ro-user@2660d5ed-e7ef-4950-a051-c84a0b4e78b0.cephfs=/ska /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/7a20f4b9d280289f509f1ad7cfa807510ea19ae34dec37b9544498fe74851ca5/globalmount -o mon_addr=deneb-dev-mon1.nubes:6789/deneb-dev-mon2.nubes:6789/deneb-dev-mon3.nubes:6789,secretfile=/tmp/csi/keys/keyfile-3710871553,ro,_netdev] stderr: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized

The not authorised is probably the culprit due to the error we see in the provisioner.

If you consider this to be a potential bug please let me know and I will raise it as such.

TIA.

[Madhu-1]

@iPraveenParihar PTAL

[iPraveenParihar]

Hey @seanmcconkey,

We're seeing two separate issues:

1. Controller Publish failure on algol-pv

W0217 16:33:22.380126       1 controllerserver.go:1169] ID: 1484 Req-ID: algol-pv controller publish secret not found: failed to decode volume ID (algol-pv): failed to decode CSI identifier, string underflow

The CSI driver is failing to decode the algol-pv volume ID because there's no check for statically provisioned volumes in the Controller(Un)PublishVolume calls. it's trying to extract a clusterID from the volume ID and failing. However, this is resolved by providing controller-publish-secret to the StorageClass.

2. MountDevice failure on deneb-pv

Warning  FailedMount  23m  kubelet  MountVolume.MountDevice failed for volume "deneb-pv" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph ro-user@2660d5ed-e7ef-4950-a051-c84a0b4e78b0.cephfs=/ska ...] stderr: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized

The mount is failing because no CephFS Metadata Server (MDS) is currently active or reachable.

Could you run the following and verify if the MDS is active?

ceph mds stat
ceph fs status
ceph -s

[seanmcconkey]

@iPraveenParihar thanks for the reply and apologies for the late response.

I am declaring the controller-publish-secret in the SC as shown above so its nothing to do with that, even though the error suggest it is.

I did figure the issue out and it was me forgetting that the fsname is not cephfs on this cluster, we overlay the name change and I was doing my tests with the un-overlayed PV template.

The only thing I'd remark on this, is why we got that error message about the secret, I can understand why we got the mds error messages though.

Thanks once again.