Kafka cluster upgradation issue

[Muddam234]

Hi All

My Kafka cluster running on Strimzi operator 0.19 and i am trying to upgrade 0.22 and when i have applied by using kubectl command the yml (strimzi-cluster-operator-0.22.1.yaml) and its successfully updated and i saw the operator running in 0.22 and when i check the Kafka cluster still using 0.19 Strimzi operator. can any one help on this issue .

operator:
.

kafka:

kubectl get crd kafkas.kafka.strimzi.io -o jsonpath="{.spec.versions[*].name}"
W0411 09:17:42.988537 8660 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
v1beta2 v1beta1 v1alpha1

operator logs:
23-04-11 07:13:02 WARN AbstractOperator:97 - Kafka resource kafka-selling in namespace selling-kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.zookeeper.metrics has been deprecated, and should now be configured using spec.zookeeper.metricsConfig. This property is removed in API version v1beta2.
2023-04-11 07:13:02 INFO AbstractOperator:254 - Reconciliation #1344(timer) Kafka(selling-kafka/kafka-selling): Kafka kafka-selling will be checked for creation or modification
2023-04-11 07:13:02 WARN AbstractOperator:97 - KafkaMirrorMaker resource kafka-selling-mirromaker-prod in namespace selling-kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.metrics has been deprecated, and should now be configured using spec.metricsConfig. This property is removed in API version v1beta2.
2023-04-11 07:13:02 WARN AbstractOperator:97 - KafkaMirrorMaker resource kafka-selling-mirromaker-prod in namespace selling-kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.metrics has been deprecated, and should now be configured using spec.metricsConfig. This property is removed in API version v1beta2.
2023-04-11 07:13:02 INFO AbstractOperator:254 - Reconciliation #1345(timer) KafkaMirrorMaker(selling-kafka/kafka-selling-mirromaker-prod): KafkaMirrorMaker kafka-selling-mirromaker-prod will be checked for creation or modification
2023-04-11 07:13:02 INFO AbstractOperator:493 - Reconciliation #1345(timer) KafkaMirrorMaker(selling-kafka/kafka-selling-mirromaker-prod): reconciled
2023-04-11 07:13:35 INFO KafkaRoller:296 - Reconciliation #1344(timer) Kafka(selling-kafka/kafka-selling): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Error getting broker config, retrying after at least 250ms
2023-04-11 07:13:35 INFO KafkaRoller:504 - Reconciliation #1344(timer) Kafka(selling-kafka/kafka-selling): Pod 0 needs to be restarted. Reason: [Pod has old generation]
2023-04-11 07:13:35 INFO KafkaRoller:296 - Reconciliation #1344(timer) Kafka(selling-kafka/kafka-selling): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-selling-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2023-04-11 07:13:35 INFO KafkaRoller:504 - Reconciliation #1344(timer) Kafka(selling-kafka/kafka-selling): Pod 2 needs to be restarted. Reason: [Pod has old generation]
2023-04-11 07:13:35 INFO KafkaAvailability:121 - __transaction_state/0 will be underreplicated (|ISR|=2 and min.insync.replicas=2) if broker 2 is restarted

I have deployed strimzi operator using these helm charts and operator is deployed successfully

[scholzj]

Please use proper formatting for the log snippets to make them more readable. I think it just says that your topics are not in sync. Normally, this means you need to wait for them to get in sync.

[Muddam234]

ok, but kafka brokers still using Strimzi operator 0.19 and i have restarted one by one but still i saw 0.19 strmzi operator.

[scholzj]

I guess you will need to elaborate on what exactly you think you see. Share the exact commands and YAML together with logs etc.

Obviously, the pods need to be rolled first to use the new container images. So for example you need to first bring your topics in-sync as mentioned above and in the log you shared. Until that is fixed and the operator can roll the pod and the pod will not use the new image.

[Muddam234]

sorry for late response

i saw the logs in strimzi operator and zookeeper pod is failing. Can you please help

2023-04-17 07:53:26 ERROR AbstractOperator:351 - Reconciliation #7441(watch) Kafka(selling-kafka/kafka-selling): Failed to update status
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://10.84.0.1/apis/kafka.strimzi.io/v1beta1/namespaces/selling-kafka/kafkas/kafka-selling/status. Message: kafkas.kafka.strimzi.io "kafka-selling" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkas, name=kafka-selling, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkas.kafka.strimzi.io "kafka-selling" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:418) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleStatusUpdate(OperationSupport.java:292) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.updateStatus(BaseOperation.java:746) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.updateStatus(BaseOperation.java:85) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
at io.strimzi.operator.common.operator.resource.CrdOperator.lambda$updateStatusAsync$2(CrdOperator.java:111) ~[io.strimzi.operator-common-0.22.1.jar:0.22.1]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
2023-04-17 07:53:26 WARN AbstractOperator:508 - Reconciliation #7441(watch) Kafka(selling-kafka/kafka-selling): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource kafka-selling-zookeeper-0 in namespace selling-kafka to be ready
at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:137) ~[io.strimzi.operator-common-0.22.1.jar:0.22.1]
at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]

[scholzj]

That suggests that your Kafka custom resource was deleted during the reconciliation. So you should check if that was the case or not.