kafka pod not coming up after deleting

[saloni-2404]

Hi Jakub,
Observing the below warnings after deleting a kafka pod and letting it come back up but it is stuck in container-creating state. What could be the issue here?

Warning FailedMount 25m kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[broker-certs client-ca-cert kafka-metrics-and-logging ready-files sedvip-qa-e-kafka-token-l5cz7 data cluster-ca]: error processing PVC sedvip-qa/data-sedvip-qa-e-kafka-0: failed to fetch PVC from API server: Get https://172.16.0.2/api/v1/namespaces/sedvip-qa/persistentvolumeclaims/data-sedvip-qa-e-kafka-0: dial tcp 172.16.0.2:443: i/o timeout
Warning FailedMount 18m (x4 over 31m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[sedvip-qa-e-kafka-token-l5cz7 data cluster-ca broker-certs client-ca-cert kafka-metrics-and-logging ready-files]: timed out waiting for the condition
Warning FailedMount 8m29s kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[cluster-ca broker-certs client-ca-cert kafka-metrics-and-logging ready-files sedvip-qa-e-kafka-token-l5cz7 data]: timed out waiting for the condition
Warning FailedMount 6m14s (x2 over 27m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[broker-certs client-ca-cert kafka-metrics-and-logging ready-files sedvip-qa-e-kafka-token-l5cz7 data cluster-ca]: timed out waiting for the condition
Warning FailedMount 3m57s (x2 over 20m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kafka-metrics-and-logging ready-files sedvip-qa-e-kafka-token-l5cz7 data cluster-ca broker-certs client-ca-cert]: timed out waiting for the condition
Warning FailedMount 102s (x2 over 10m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[ready-files sedvip-qa-e-kafka-token-l5cz7 data cluster-ca broker-certs client-ca-cert kafka-metrics-and-logging]: timed out waiting for the condition

[scholzj]

As the event says, the pod requires the data volume, but it is failing to mount because of error processing PVC sedvip-qa/data-sedvip-qa-e-kafka-0: failed to fetch PVC from API server: Get https://172.16.0.2/api/v1/namespaces/sedvip-qa/persistentvolumeclaims/data-sedvip-qa-e-kafka-0: dial tcp 172.16.0.2:443: i/o timeout.

I think you would need to look into your infrastructure why the volume does not mount.

[saloni-2404]

ok thanks turns out it was an issue with GKE

[shaypi]

Hi @saloni-2404Can you please explain what was the issue in the GKE