[Bug]: Getting error io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem while upgrading to 3.4.0 Kafka

[kiranmenon]

Bug Description

I was trying to upgrade strimzi (from 0.32.0 -->0.34.0) and then kafka (from 3.3.1 --> 3.4.0).
strimzi-0.34.0-logs.txt
Then, getting the error as shown in the attachment. It says "o.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod fabric-kafka-0 cannot be updated right now."

Steps to reproduce

1. Deploy strimzi 0.32.0 and create a kafka 3.3.1
   kafka-0.32.0.yml.txt
2. Now upgrade strimzi to 0.34.0. Verify strimzi operator is ready
3. Now upgrade kafka to 3.4.0
   kafka-0.34.0.yml.txt

Expected behavior

Kafka upgrade to 3.4.0 should be successful

Strimzi version

0.34.0

Kubernetes version

1.25.4

Installation method

YAML files

Infrastructure

OCI

Configuration files and logs

No response

Additional context

No response

[scholzj]

The error says the following:

2023-04-11 10:38:45 INFO  KafkaAvailability:135 - Reconciliation #85(timer) Kafka(kafka/fabric): __strimzi_store_topic/0 will be under-replicated (ISR={0}, replicas=[0,1], min.insync.replicas=1) if broker 0 is restarted.

That means that this topic is not in-sync and that is why the broker 0 cannot be restarted. You have to wait until it gets up to sync and then the pod will be rolled.

I don't think this is in any way related to the upgrade.

[scholzj]

Triaged on 20.4.2023: Seems more like a question than a bug. Should be converted to discussion for the time being.

[Sakib37]

Also having similar error in a kafka which has 0 producer and 0 consumer at this moment.
error:

 __strimzi_store_topic/0 will be under-replicated (ISR={1,0}, replicas=[0,6,1], min.insync.replicas=2) if broker 1 is restarted

That means that this topic is not in-sync and that is why the broker 0 cannot be restarted. You have to wait until it gets up to sync and then the pod will be rolled.

The problem is, this topic never(waited more than 8 hours) gets synced and operator blocks new broker to be created and join the cluster. I tried to reduce min.insync.replicas to 1 so that operator allows to restart broker 1 in my case. But seems like it is not possible to change the min.insync.replicas or the new config is not being active.

For me, the cluster is stuck with this error message for last 8 hours without any consumer or producer.
@scholzj could you please suggest any workaround to overcome from this situation?

[scholzj]

So, you scaled down the cluster by mistake? Or how was the broker 6 deleted?

[Sakib37]

yes, scaled down the cluster by mistake. so 2 broker (broker-5 and broker-6 ) got deleted. But I was using the following config

    num.partitions: 6   
    default.replication.factor: 5
    offsets.topic.replication.factor: 5
    transaction.state.log.replication.factor: 5
    transaction.state.log.min.isr: 2

    min.insync.replicas: 2

So with this config, I should be able to recover if 2 brokers got deleted by mistake. However, it seems like topic __strimzi_store_topic ignored the default.replication.factor configuration and had only 3 ISR. Not sure why it ignored that default.replication.factor though.
However, all the topics I created, followed default.replication.factor and had 5 replicas(for partitions) and I believe those topic (with 5 replication factor) will recover without any issue if I could brought back broker 5 and broker 6. But unfortunately, strimzi operator does not allow this due to the above error.

[scholzj]

TBH, replication factor of 5 makes sense only very rarely. I'm not sure I understand the motivation for that.

---

The easiest way to fix it is to recover the missing brokers. If you use Stirmzi 0.35, I think you should need to only change back the .spec.kafka.replicas. If you use some older version, you might need to do non-cascading delete of the Kafka StatefulSet / StrimziPodSet (depends on the version you use and the settings). Then in the next reconciliation the two new pods should be started. And htne you need to wait for the brokers to recover the data they are missing.

[Sakib37]

The only motivation is to allow reliability even if broker(pod) goes down due to unplanned Kubernetes node termination. Previously with 3 replication factor faced the same issue and strimzi operator did not allow the deleted broker to rejoin the cluster. So increased the replication factor to 5 and kept min.insync.replicas to 2 so that even if two brokers go down strimzi will not object that topic will be under-replicated

Updated .spec.kafka.replicas with strimzi 0.34 but got the above error. However, as you have already mentioned, it will work only with 0.35. I will try to figure out how can I do this non-cascading delete of StrimziPodSet.

[scholzj]

Previously with 3 replication factor faced the same issue and strimzi operator did not allow the deleted broker to rejoin the cluster. So increased the replication factor to 5 and kept min.insync.replicas to 2 so that even if two brokers go down strimzi will not object that topic will be under-replicated

I'm not really sure what exactly does that mean to be honest. The common settings are 3 replicas with at least 2 in-sync. Having 5 replicas is expensive and outside of some niche use cases does not really add anything. There are plenty of users using Strimzi with that setting without any problems.

I will try to figure out how can I do this non-cascading delete of StrimziPodSet.

You can do it with kubectl delete => non-cascading means that you should use the option --cascade=orphan