How to remove min.insync.replicas from topics?

[wishkres]

Hello,

While troubleshooting #7016, I added min.insync.replicas: 2 to my cluster configuration. I was able to add and remove the setting several times through my kubectl configuration while debugging, but now the setting has somehow gotten applied to my topics and I can't figure out how to get rid of it.

Kubernetes is reporting that all of my KafkaTopics contain:

spec:
  config:
    min.insync.replicas: "2"

I have tried deleting that line through both the Rancher UI and kubectl, but it keeps coming back.

I have also tried to bypass Kubernetes and remove the config from the topic itself by running the following command from the tls-sidecar:

bin/kafka-configs.sh --bootstrap-server BOOTSTRAP --command-config CONFIG_FILE --alter --entity-type topic --entity-name __strimzi_store_topic --delete-config min.insync.replicas

And although the command returns "Completed updating config for topic __strimzi_store_topic" nothing appears to be changing.

All my topics are back to being stuck in an InvalidStateException and in a non-ready state because replicas are set to 1 and min.insync.replicas is set to 2, and I can't figure out how to get it back to normal.