svc resourceVersion keeps updating every 2 mins

[pupseba]

Hi,

our kafka services for some reason, keep changing their resourceVersion every 2 minutes exactly.

kafka-strimzi-kafka-external-0 LoadBalancer 172.17.251.51 9093:30539/TCP
kafka-strimzi-kafka-external-1 LoadBalancer 172.17.238.41 9093:30247/TCP
kafka-strimzi-kafka-external-2 LoadBalancer 172.17.226.93 9093:32757/TCP
kafka-strimzi-kafka-external-3 LoadBalancer 172.17.230.120 9093:30185/TCP
kafka-strimzi-kafka-external-bootstrap LoadBalancer 172.17.239.139 9093:31150/TCP
kafka-strimzi-kafka-extplain-0 LoadBalancer 172.17.226.61 9095:31352/TCP
kafka-strimzi-kafka-extplain-1 LoadBalancer 172.17.236.172 9095:31349/TCP
kafka-strimzi-kafka-extplain-2 LoadBalancer 172.17.239.83 9095:32557/TCP
kafka-strimzi-kafka-extplain-3 LoadBalancer 172.17.231.64 9095:31073/TCP
kafka-strimzi-kafka-extplain-bootstrap LoadBalancer 172.17.243.28 9095:30969/TCP

Nothing changes in the services itself exept for the resourceVersion. This, makes other services in the cluster that have "watches" over these resources (like metalLB), to react unnecessarily which generates some problems for us.

Would you know any reason why the kafka-operator reconcile would trigger such action if, as far as I know, there is no change in those services definitions?

Thank you,
Seb

[scholzj]

There is likely something in your cluster changing the service which triggers back the operator to change it again. You should probably get the operator logs in a DEBUG level - that might tell us some more details and what is changing.

[pupseba]

Can't change the verbosity of logs at the moment, but I was able to test a theory and it seems that MetalLB watches over services are "fighitng over" kafka services with kafka-operator. In particular, MetalLB adds this annotation: metallb.universe.tf/ip-allocated-from-pool and by doing that, increases the resourceVersion of the service. Then, every 2 minutes, the reconcile loop of kafka-operator kicks in and removes that annotation and then again, increases the resourceVersion.

Is it possible to tell kafka-operator to ignore that annotation? or would it be maybe any other way to handle this?

[scholzj]

What is its value? If the value is fixed, you can set it in the .spec.kafka.template to avoid this.

[pupseba]

Thought about it, but I'm unsure how that would affect MetalLB itself. Checking now:
https://github.com/metallb/metallb/blob/c0f0ee297c6b5ab19feacb6e7e0598cc051f4069/controller/service.go#L32

We actually pass to the template this other annotation: metallb.universe.tf/address-pool: dynamic-pool-1, so the value (dynamic-pool-1) would be quite easy to template too.

I'm checking now how this would affect MetalLB itself: https://github.com/metallb/metallb/blob/c0f0ee297c6b5ab19feacb6e7e0598cc051f4069/controller/service.go#L32

Would be faster to test, but I will not be able to test until Monday though. Will report back here.

Thank you!

[scholzj]

If the annotation has always the same fixed value it should be easy and it would basically tell Strimzi to see this annotation as expected. So it sounds like it might work here for you. It would be a bit more complicated if it would be something different on each service and with some more or less random values. Anyway, give it a try when you have the time.

[pupseba]

A colleague of mine just found this open issue: #6938

I will be testing hardcoding the annotation of course, hope it does not mess up too much with MetalLB.

[scholzj]

Yes, that issue covers the problematic of labels and annotations.

[pupseba]

Hi scholzj,

I just left a comment in the existing issue, which is specific for EKS but as you mentioned, it covers the problematic affecting MetalLB too.

Right now, we have MetalLB and strimzi fighting over those resources, since both increase the resourceVersion (metallb once it adds the annotation and kafka-operator every 2 minutes by removing it).

Thank you for your help!