pvc failed to connect to volume

[wangshuai0709]

My Rook cluster and PVC are in different Kubernetes clusters.My PVC is for cross-cluster access with Rook.
ceph-csi-rbd-provisioner-7bf85c4477-ljqc4 pod log:

I0626 02:34:20.294984       1 utils.go:265] ID: 1733 GRPC call: /csi.v1.Identity/Probe
I0626 02:34:20.295032       1 utils.go:266] ID: 1733 GRPC request: {}
I0626 02:34:20.295056       1 utils.go:272] ID: 1733 GRPC response: {}
I0626 02:34:35.527401       1 utils.go:298] ID: 1728 Req-ID: pvc-712f4a33-0165-46d8-a295-af804ccf2095 Slow GRPC call /csi.v1.Controller/CreateVolume (5m0s)
I0626 02:34:35.527499       1 utils.go:265] ID: 1734 Req-ID: pvc-712f4a33-0165-46d8-a295-af804ccf2095 GRPC call: /csi.v1.Controller/CreateVolume
I0626 02:34:35.527561       1 utils.go:300] ID: 1728 Req-ID: pvc-712f4a33-0165-46d8-a295-af804ccf2095 Slow GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-712f4a33-0165-46d8-a295-af804ccf2095","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-712f4a33-0165-46d8-a295-af804ccf2095","csi.storage.k8s.io/pvc/name":"rbd-fox-dev","csi.storage.k8s.io/pvc/namespace":"rook-ceph","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
I0626 02:34:35.527570       1 utils.go:266] ID: 1734 Req-ID: pvc-712f4a33-0165-46d8-a295-af804ccf2095 GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-712f4a33-0165-46d8-a295-af804ccf2095","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-712f4a33-0165-46d8-a295-af804ccf2095","csi.storage.k8s.io/pvc/name":"rbd-fox-dev","csi.storage.k8s.io/pvc/namespace":"rook-ceph","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
I0626 02:34:35.527834       1 rbd_util.go:1410] ID: 1734 Req-ID: pvc-712f4a33-0165-46d8-a295-af804ccf2095 setting disableInUseChecks: false image features: [] mounter: rbd
E0626 02:34:35.536417       1 controllerserver.go:243] ID: 1728 Req-ID: pvc-712f4a33-0165-46d8-a295-af804ccf2095 failed to connect to volume : failed to get connection: connecting failed: rados: ret=-110, Connection timed out
E0626 02:34:35.536533       1 utils.go:270] ID: 1728 Req-ID: pvc-712f4a33-0165-46d8-a295-af804ccf2095 GRPC error: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-110, Connection timed out

run ceph commands IN ceph-csi-rbd-provisioner-7bf85c4477-kshp8 pod. Result :

/etc/ceph/keyring:

[client.admin]
    key = AQCs81hogE7mBBAA7Weq07UPC7GL9pTcR22XSQ==
    caps mds = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"
    caps mgr = "allow *"

The rook cluster keyring:

I dont known why not connect to mon

[Madhu-1]

@wangshuai0709 is this same as #5391? if yes, i would suggest to close this as duplicate.

[wangshuai0709]

OK,Thanks.