Strimzi Operator failing with " Exception occurred while renewing lock: Unable to update LeaseLock"

[sreejesh-radhakrishnan-db]

Operator - 0.34
KAFKA - 3.4
GKE - 1.26
we use STATEFULSET (instead of strimzi pod set) to run the KAFKA broker and ZK

On install everything works and KAFKA is up , Operator is able to reconcile and everything works.

On the first certifcare renewal of KAFKA- we have observed Operator fails to reconcile. the DEBUG looks something like below

2023-07-10 11:45:09 DEBUG Config:854 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2023-07-10 11:45:09 DEBUG Config:849 - Trying to configure client namespace from Kubernetes service account namespace path...
2023-07-10 11:45:09 DEBUG Config:558 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2023-07-10 11:45:09 DEBUG Config:551 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt}].
2023-07-10 11:45:09 DEBUG Config:545 - Found service account host and port: 240.13.132.129:443
2023-07-10 11:45:09 DEBUG Config:535 - Trying to configure client from service account...
2023-07-10 11:45:09 DEBUG Config:636 - Did not find Kubernetes config at: [/home/strimzi/.kube/config]. Ignoring.
2023-07-10 11:45:09 DEBUG Config:630 - Trying to configure client from Kubernetes config...
"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException: Unable to update LeaseLock
at io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LeaseLock.update(LeaseLock.java:102) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:214) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$renewWithTimeout$6(LeaderElector.java:175) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$loop$8(LeaderElector.java:258) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [replace] for kind: [Lease] with name: [strimzi-cluster-operator] in namespace: [kafka] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$replace$0(HasMetadataOperation.java:169) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:172) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:113) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:41) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.replace(BaseOperation.java:1077) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.replace(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LeaseLock.update(LeaseLock.java:100) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
... 7 more
Caused by: java.io.IOException: /240.13.160.28:34738: GOAWAY received
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:533) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleUpdate(OperationSupport.java:366) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleUpdate(BaseOperation.java:709) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$replace$0(HasMetadataOperation.java:167) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:172) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:113) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:41) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.replace(BaseOperation.java:1077) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.replace(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LeaseLock.update(LeaseLock.java:100) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
... 7 more
Caused by: java.io.IOException: /240.13.160.28:34738: GOAWAY received
at jdk.internal.net.http.Http2Connection.handleGoAway(Http2Connection.java:1008) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection.handleConnectionFrame(Http2Connection.java:873) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection.processFrame(Http2Connection.java:748) ~[java.net.http:?]
at jdk.internal.net.http.frame.FramesDecoder.decode(FramesDecoder.java:155) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection$FramesController.processReceivedData(Http2Connection.java:232) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection.asyncReceive(Http2Connection.java:674) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.processQueue(Http2Connection.java:1310) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.runOrSchedule(Http2Connection.java:1328) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1354) ~[java.net.http:?]
at jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1288) ~[java.net.http:?]
at jdk.internal.net.http.common.SSLTube$DelegateWrapper.onNext(SSLTube.java:210) ~[java.net.http:?]
at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:492) ~[java.net.http:?]
at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:295) ~[java.net.http:?]
at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:316) ~[java.net.http:?]
at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:259) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
at jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:232) ~[java.net.http:?]
at jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:198) ~[java.net.http:?]
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:444) ~[java.net.http:?]
at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:268) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
... 3 more"
2023-07-10 11:45:01 DEBUG LeaderElector:182 - Exception occurred while renewing lock: Unable to update LeaseLock

[scholzj]

I think you would need to share a full log. But from what you shared it looks like some issue communicating with your Kubernetes API.