Kafkaconnect Deployment max surge and max unavailable

[ee07dazn]

Hi
Wish you a very Happy New Year!!

As seen above, my default Kafkaconnect CRD creates a deployment with RollingUpdateStrategy: 0 max unavailable, 1 max surge. I would like to change this but on the docs I only see the option to specify the strategy.

Context - I am using reloader to detect change in configmap and/or secrets to roll over the pods but this is not working as desired, for > 1 replica. I believe changing the above should help sort this issue for me.

[scholzj]

The docs are correct, you cannot configure it. I'm not sure I understand why would the reloader have problems with it. This is IMHO a regular configuration for a Deployment. In any case, the plan is to move away from using Deployments since they do not work well for Kafka Connect (see https://github.com/strimzi/proposals/blob/main/045-Stable-identities-for-Kafka-Connect-worker-nodes.md for more details).

[ee07dazn]

Thanks for sharing the proposal.

I tested the reloader with the same settings on a different sample deployment and it works perfectly alright there. So something seems to be up on my strimzi Kafkaconnect.

The strange behaviour I am noticing is
when a configmap changes, reloader correctly correct triggers the deployment to rollover the pods
However,

1. the pods as well as the replicaset starts to rollover and then the pods come in running state.
2. but soon after, the deployment takes out the new replicaset (*ddf in image) and brings back the old replicaset (*b57 in image)

Any idea what might be going on here ?

I have tried to look at the events and other things but i am not able to get a root cause of this issue.

[ee07dazn]

Ahhkk... I guess in some cases where the connector is not as much impacted, we should be good, right ?For ex- when changing the configMap for jmxprom etc.

On that connector restart point, one of the old version docs had mentioned about annotations that should trigger a connector/task restart but that seems to have gone away from the more recent docs.

I had also tried these annotations but they didn't work. Can you please confirm they are still valid ?

Kubectl annotate kafkaconnector strimzi.io/restart=true

Kubectl annotate kafkaconnector Strimzi.io/restart-task=

[scholzj]

The operator IMHO should already today take care of any changes in the secrets it uses directly. The Prometheus metrics configuration is actually reloaded without restart. If for example some secret with password or certificate used for Kafka authentication changes, it will roll Connect on its own. But it will not do that for the secrets or config maps you mount for connectors.

The annotations should IMHO still work. If they don't, it would be a bug. But keep in mind that you need to have the Connector operator enabled and use the KafkaConnector resources. Otherwise, you need to restart them through the Connect REST API.

[ee07dazn]

i am definitely not using the REST API and only using the operator along with the resource. I will double check and revert back.

[ee07dazn]

With the mounts, as per Kubernetes doc from 1.21 onwards, when a CM is mounted as volume (not when used as environment variable), and it changes, it get auto updated but i need to check for secrets.

AFAIK the Secrets update as well. but the problem is that Connect does not care about it. What the rolling of the pods does for you is not just update the ConfigMap or Secret content in the Pod. It also has to essentially restart the connectors and that is when they reload the data. So you would still need to trigger the connectors to reload the data somehow (e.g. restart them I guess).

Hi
So on that comment where you say you need to restart the connector possibly, this is specific when something that is used by the connector (say a secret) has been updated in the pod, which rolled over, right? otherwise the connector still uses the old config (say that secret) ?

I tried making change to

1. a running connector
2. rolling over pods in general
   and in both scenarios, the rebalancing happened and the connector worked fine eventually in about couple of mins, and whilst that was happening, my data was still coming through correctly. So all in all, the transition seemed smooth.

[scholzj]

When you roll the pod, the connectors (their tasks) start again. So yes, in that case they load any new changes automatically. But they would not do that if the pod doesn't restart.