Failure executing patch after upgrade from 0.30.0 to 0.31.1

[eric-aops]

Hi,
I've upgraded Strimzi Kafka Operator but the reconciliation fails, the Pods don't get restarted at all.
Below is the error

2022-10-10 05:58:19 WARN  AbstractOperator:516 - Reconciliation #4(timer) Kafka(dashur-data/kafka): Failed to reconcile
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.211.64.1/api/v1/namespaces/dashur-data/services/kafka-zookeeper-client. Message: the server rejected our request due to an error in our request. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=the server rejected our request due to an error in our request, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:713) ~[io.fabric8.kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:693) ~[io.fabric8.kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:644) ~[io.fabric8.kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:581) ~[io.fabric8.kubernetes-client-6.1.1.jar:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:622) ~[io.fabric8.kubernetes-client-6.1.1.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$4.onResponse(OkHttpClientImpl.java:268) ~[io.fabric8.kubernetes-httpclient-okhttp-6.1.1.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

Anybody ran into this one already? I have no clue on what might be wrong.

Kind regards

Eric

[scholzj]

might be similar to #7442. But I didn't managed to reproduce it.

[eric-aops]

Hi @scholzj ,

Thanks for your answer. It wasn't identical to that issue but it pointed me in the right direction. I still haven't found the root cause but did find a workaround that allowed the upgrade to proceed.
I have Strimzi running on 5 clusters, all the same version of Kubernetes, all the same version of Strimzi and the same upgrade path.
On the DEV clusters no issues, on one of the STG clusters no issues and the other one the indicated error/warning.

I couldn't remove the internalTrafficPolicy as indicated in the patch, but the next thing I did was change it from Cluster to Local. Then the reconciliation worked as expected and once it was done, the value for the internalTrafficPolicy was automatically reset to Cluster.

The weird thing is that this only happened on one cluster and I can't find any discrepancies...

Just mentioning what worked for me, hope it helps someone down the line if they run into the same or similar issue.

Kind regards,

Eric

[scholzj]

Thanks for mentioning what helped you. This seems like a real weird issue :-/.