Volume Resize Failure (Azure)

[siegenthalerroger]

Hi there, I was required to increase the storage space of our three node kafka cluster and ran into an issue on one of the nodes. Two nodes completed the resize with no issue (no kafka restarts, no special events), the third one however ran into a FileSystemResizePending error.

From my understanding, strimzi is supposed to roll the cluster in this case, to facilitate the required (re-)start of the pod. This did not happen and I started seeing errors in the strimzi logs. It seems like strimzi was just waiting for the pod to be ready and not realising that it needed to restart the pod to finalise the resize. In a different discussion it seems like strimzi did actually do the pod roll, so I think the issue may be related to the "race condition" that was encountered on the CSI resizer (see PVC Events).

Sidenote: After a good while (>30mins), I decided to delete the offending PVC and then strimzi successfully created a new one and kafka rebuilt itself. Had this not worked I would have restored the backup from before the resize.

This was with strimzi version 0.35.0 on Azure AKS.

My actual questions now as follows:

1. Is it possible, that the PVC error event is what made the PVC require a restart to finish the resize (vs the other nodes that just resized transparently)?
2. Is it strimzi's or the azure CSI driver's responsibility to actually deal with the error and retry the annotation setting?
3. Is a strimzi bug issue warranted, as I feel strimzi should have been capable of restarting the pod?

PVC Resize

Status Conditions:

  conditions:
    - type: FileSystemResizePending
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-06-25T10:53:14Z'
      message: >-
        Waiting for user to (re-)start a pod to finish file system resize of
        volume on node.

Events:

Strimzi Logs

Note that the PVC resize only begins after the first exception (2023-06-25 10:49:40 INFO PvcReconciler:132).

2023-06-25 10:45:38 INFO  AbstractOperator:239 - Reconciliation #15918(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2023-06-25 10:45:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:45:41 INFO  AbstractOperator:510 - Reconciliation #15918(timer) Kafka(kafka/kafka-cluster): reconciled
2023-06-25 10:47:38 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-06-25 10:47:38 INFO  AbstractOperator:239 - Reconciliation #15919(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2023-06-25 10:47:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:47:41 INFO  AbstractOperator:510 - Reconciliation #15919(timer) Kafka(kafka/kafka-cluster): reconciled
2023-06-25 10:47:48 ERROR AbstractOperator:492 - Watcher closed with exception in namespace kafka-operator
io.fabric8.kubernetes.client.WatcherException: Could not process websocket message
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:462) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.net.SocketException: Connection reset
	at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
	at jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1170) ~[java.net.http:?]
	at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833) ~[java.net.http:?]
	at jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
	at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:774) ~[java.net.http:?]
	at jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:957) ~[java.net.http:?]
	at jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253) ~[java.net.http:?]
	at jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:979) ~[java.net.http:?]
	at jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:934) ~[java.net.http:?]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:934) ~[java.net.http:?]
2023-06-25 10:49:38 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-06-25 10:49:38 INFO  AbstractOperator:239 - Reconciliation #15920(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2023-06-25 10:49:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:49:40 INFO  PvcReconciler:132 - Reconciliation #15920(timer) Kafka(kafka/kafka-cluster): Resizing PVC data-kafka-cluster-kafka-0 from 100 to 150Gi.
2023-06-25 10:49:40 INFO  PvcReconciler:132 - Reconciliation #15920(timer) Kafka(kafka/kafka-cluster): Resizing PVC data-kafka-cluster-kafka-1 from 100 to 150Gi.
2023-06-25 10:49:40 INFO  PvcReconciler:132 - Reconciliation #15920(timer) Kafka(kafka/kafka-cluster): Resizing PVC data-kafka-cluster-kafka-2 from 100 to 150Gi.
2023-06-25 10:49:41 INFO  CrdOperator:133 - Reconciliation #15920(timer) Kafka(kafka/kafka-cluster): Status of Kafka kafka-cluster in namespace kafka has been updated
2023-06-25 10:49:41 INFO  AbstractOperator:510 - Reconciliation #15920(timer) Kafka(kafka/kafka-cluster): reconciled
2023-06-25 10:50:35 INFO  StrimziPodSetController:313 - Reconciliation #15921(watch) StrimziPodSet(kafka/kafka-cluster-kafka): StrimziPodSet will be reconciled
2023-06-25 10:50:35 INFO  StrimziPodSetController:349 - Reconciliation #15921(watch) StrimziPodSet(kafka/kafka-cluster-kafka): reconciled
2023-06-25 10:50:35 INFO  StrimziPodSetController:313 - Reconciliation #15922(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): StrimziPodSet will be reconciled
2023-06-25 10:50:35 INFO  StrimziPodSetController:349 - Reconciliation #15922(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): reconciled
2023-06-25 10:51:38 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-06-25 10:51:38 INFO  AbstractOperator:239 - Reconciliation #15923(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2023-06-25 10:51:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:51:41 INFO  AbstractOperator:510 - Reconciliation #15923(timer) Kafka(kafka/kafka-cluster): reconciled
2023-06-25 10:53:38 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-06-25 10:53:38 INFO  AbstractOperator:239 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2023-06-25 10:53:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:53:40 INFO  PvcReconciler:83 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): The PVC data-kafka-cluster-kafka-1 is waiting for file system resizing and the pod kafka-cluster-kafka-1 needs to be restarted.
2023-06-25 10:53:40 INFO  KafkaRoller:551 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Rolling Pod kafka-cluster-kafka-1/1 due to [File system needs to be resized]
2023-06-25 10:53:40 INFO  PodOperator:54 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Rolling pod kafka-cluster-kafka-1
2023-06-25 10:54:38 INFO  AbstractOperator:380 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Reconciliation is in progress
2023-06-25 10:55:35 INFO  StrimziPodSetController:313 - Reconciliation #15925(watch) StrimziPodSet(kafka/kafka-cluster-kafka): StrimziPodSet will be reconciled
2023-06-25 10:55:35 INFO  StrimziPodSetController:349 - Reconciliation #15925(watch) StrimziPodSet(kafka/kafka-cluster-kafka): reconciled
2023-06-25 10:55:35 INFO  StrimziPodSetController:313 - Reconciliation #15926(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): StrimziPodSet will be reconciled
2023-06-25 10:55:35 INFO  StrimziPodSetController:349 - Reconciliation #15926(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): reconciled
2023-06-25 10:55:35 INFO  StrimziPodSetController:313 - Reconciliation #15927(watch) StrimziPodSet(kafka/kafka-cluster-kafka): StrimziPodSet will be reconciled
2023-06-25 10:55:35 INFO  StrimziPodSetController:349 - Reconciliation #15927(watch) StrimziPodSet(kafka/kafka-cluster-kafka): reconciled
2023-06-25 10:55:38 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-06-25 10:55:38 INFO  AbstractOperator:380 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Reconciliation is in progress
2023-06-25 10:55:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:56:38 INFO  AbstractOperator:380 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Reconciliation is in progress
2023-06-25 10:57:38 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-06-25 10:57:38 INFO  AbstractOperator:380 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Reconciliation is in progress
2023-06-25 10:57:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:58:38 INFO  AbstractOperator:380 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Reconciliation is in progress
2023-06-25 10:58:47 INFO  KafkaRoller:313 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Could not verify pod kafka-cluster-kafka-1/1 is up-to-date, giving up after 10 attempts. Total delay between attempts 127750ms
io.strimzi.operator.cluster.operator.resource.KafkaRoller$FatalProblem: Error while waiting for restarted pod kafka-cluster-kafka-1 to become ready
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$awaitReadiness$17(KafkaRoller.java:706) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.await(KafkaRoller.java:739) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.awaitReadiness(KafkaRoller.java:706) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartAndAwaitReadiness(KafkaRoller.java:700) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:394) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:307) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095) ~[?:?]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.await(KafkaRoller.java:735) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	... 10 more
2023-06-25 10:58:47 ERROR AbstractOperator:260 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): createOrUpdate failed
io.strimzi.operator.cluster.operator.resource.KafkaRoller$FatalProblem: Error while waiting for restarted pod kafka-cluster-kafka-1 to become ready
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$awaitReadiness$17(KafkaRoller.java:706) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.await(KafkaRoller.java:739) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.awaitReadiness(KafkaRoller.java:706) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartAndAwaitReadiness(KafkaRoller.java:700) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:394) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:307) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095) ~[?:?]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.await(KafkaRoller.java:735) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	... 10 more
2023-06-25 10:58:47 ERROR Util:166 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Exceeded timeout of 300000ms while waiting for Pods resource kafka-cluster-kafka-1 in namespace kafka to be ready
2023-06-25 10:58:47 WARN  KafkaRoller:863 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Error waiting for pod kafka/kafka-cluster-kafka-1 to become ready: io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource kafka-cluster-kafka-1 in namespace kafka to be ready
2023-06-25 10:58:47 INFO  CrdOperator:133 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Status of Kafka kafka-cluster in namespace kafka has been updated
2023-06-25 10:58:47 WARN  AbstractOperator:525 - Reconciliation #15924(timer) Kafka(kafka/kafka-cluster): Failed to reconcile
io.strimzi.operator.cluster.operator.resource.KafkaRoller$FatalProblem: Error while waiting for restarted pod kafka-cluster-kafka-1 to become ready
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$awaitReadiness$17(KafkaRoller.java:706) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.await(KafkaRoller.java:739) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.awaitReadiness(KafkaRoller.java:706) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartAndAwaitReadiness(KafkaRoller.java:700) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:394) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:307) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095) ~[?:?]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.await(KafkaRoller.java:735) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
	... 10 more
2023-06-25 10:59:38 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-06-25 10:59:38 INFO  AbstractOperator:239 - Reconciliation #15930(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2023-06-25 10:59:39 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-operator
2023-06-25 10:59:40 INFO  PvcReconciler:83 - Reconciliation #15930(timer) Kafka(kafka/kafka-cluster): The PVC data-kafka-cluster-kafka-1 is waiting for file system resizing and the pod kafka-cluster-kafka-1 needs to be restarted.
2023-06-25 10:59:40 WARN  ClientUtils:75 - Couldn't resolve server kafka-cluster-kafka-1.kafka-cluster-kafka-brokers.kafka.svc.cluster.local:9091 from bootstrap.servers as DNS resolution failed for kafka-cluster-kafka-1.kafka-cluster-kafka-brokers.kafka.svc.cluster.local
2023-06-25 11:00:35 INFO  StrimziPodSetController:313 - Reconciliation #15931(watch) StrimziPodSet(kafka/kafka-cluster-kafka): StrimziPodSet will be reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:349 - Reconciliation #15931(watch) StrimziPodSet(kafka/kafka-cluster-kafka): reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:313 - Reconciliation #15932(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): StrimziPodSet will be reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:349 - Reconciliation #15932(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:313 - Reconciliation #15933(watch) StrimziPodSet(kafka/kafka-cluster-kafka): StrimziPodSet will be reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:349 - Reconciliation #15933(watch) StrimziPodSet(kafka/kafka-cluster-kafka): reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:313 - Reconciliation #15934(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): StrimziPodSet will be reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:349 - Reconciliation #15934(watch) StrimziPodSet(kafka/kafka-cluster-zookeeper): reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:313 - Reconciliation #15935(watch) StrimziPodSet(kafka/kafka-cluster-kafka): StrimziPodSet will be reconciled
2023-06-25 11:00:35 INFO  StrimziPodSetController:349 - Reconciliation #15935(watch) StrimziPodSet(kafka/kafka-cluster-kafka): reconciled
2023-06-25 11:00:38 INFO  AbstractOperator:380 - Reconciliation #15930(timer) Kafka(kafka/kafka-cluster): Reconciliation is in progress

[scholzj]

I don't think your log makes it really clear what happened in your case. Strimzi will restart the pods when they get into the FileSystemresizePending state. But it also needs to make sure that the Kafka cluster is available etc. So no, it will not restart the pod if it was not ready. But you do not explain anywhere why was it not ready. So it is not clear what was happening in your cluster.

Also, based on previous discussion, Azure has some special behaviors when it comes to resizing. In the past it was reported that it does not use the FileSystemresizePending state and that restarting the pod is not always enough as that does not lead to it being always unmounted from the node. This is something we have only limited control over.

[siegenthalerroger]

I may not have made it clear but the cluster was available and running in the (partially) failed state. With two nodes at 150Gi and one in some weird state all three pods were running and our kafka monitoring was showing messages being produced and consumed normally.

Is there anywhere where I could see whatever made strimzi believe the cluster was not available?

[scholzj]

I think you are mixing various things. The cluster might have been available, but the pods were clearly not all ready and running according to the log.

[siegenthalerroger]

Ah you are right, the kafka-1 node was down apparently. Unsure why I thought otherwise.

So the assumed case is that strimzi did infact attempt the restart, this failed (due to something azure internal), leading to strimzi then not knowing what to do anymore?

[scholzj]

I do not have enough information to assume anything from that TBH.