Understanding KafkaConnect "ConfigMaps" under the hood

[cstmgl]

Hi all,

I have a discussion that is related to this issue more or less.
#5938

Basically every once in a while when upgrading my Kafka Connect the process is not smooth and gets stuck.
An infinite reconciliation loop that requires me to scale down the Strimzi Cluster Operator and then after the scale up everything goes ok.

Now today we got a new sort of error that was fixed the same way but really gets me puzzled because I do not get it.

So my KafkaConnect resource is very simple and it references a docker image I pre-build (so I don't use the functionality that allow us to build images based on the plugins we want to insert into it).
Anyway today when we upgraded our cluster we go this error:

Operation: [get] for kind: [ConfigMap] with name:
[%KafkaConnectName%-dockerfile] in namespace: 
[my-namespace] failed.
reason: KubernetesClientException

What I'm trying to understand is why is Strimzi trying to find a config map with the same name as the kafka connect instance and it fails.
The part that I'm curious is that after restarting the strimzi cluster operator the error goes away.
Also in any of my environments I've never seen a configmap with that pattern of a name, even when things are working, so I'm a bit curious to know how this works, if someone knows how to explain it.

Thank you for any feedback.

Miguel

[scholzj]

I don't understand the first part about the infinite loop. I'm not aware of any such issue. So you would need to provide more details.

For the second part, without knowing the exception, it is hard to tell the cause. It could have been just some glitch in your Kube cluster for example if it disappeared after a restart. In general, if you do not use the Connect Build feature, it will probably check if the config map with the generated Dockerfile needs to be deleted after some previous use.

[cstmgl]

Sorry I totally forgot about this.
So the experienced I observed is that when the KafkaConnect get's deleted (for example in an update) the Strimzi Cluster Operator get's stuck in an infinite reconciliation (I guess because the old pods disappear and the new ones never get created).
Kind of like explained by the other discussion #5938

I noticed that the reconciliation stays in that status for more than a day or so, eventually we have to delete the Strimzi Cluster Operator pod and let it recreate and go back to normal.

[scholzj]

If you could provide a debug log from the operator, then there might be something to improve. Nothing should really take days - the operations have normally timeout of 5 minutes. So after 5 minutes it should fail and if the resource was deleted, that should be it.