StrimziPodSet don't re-scale kafka pod after draining node

[Rabzou]

Hi Strimzi team,
I'm using Strimzi 0.34.0 with kafka 3.4.0.
I try to drain node and test the scheduling of kafka pods.
So with StatefulSet, (by disabling StrimziPodSet feature gate) i have no problem. The kafka pod is correctly delete, and scaled up after the drain.

But with StrimziPodSet, the kafka pod is well deleted, but it don't re-scale.
I get this error in the operator logs:

2023-04-26 15:48:14 ERROR StrimziPodSetController:374 - Reconciliation #906(watch) StrimziPodSet(brice-project/myoffice-myapp-strimzi-kafka): StrimziPodSet myoffice-myapp-strimzi-kafka in namespace brice-project reconciliation failed
12718io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1:443/api/v1/namespaces/brice-project/pods. Message: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": failed to call webhook: Post "https://istiod-basic-install.brice-project-istio.svc:443/inject?timeout=10s": no endpoints available for service "istiod-basic-install". Received status: Status(apiVersion=v1, code=500, details=StatusDetails(causes=[StatusCause(field=null, message=failed calling webhook "sidecar-injector.istio.io": failed to call webhook: Post "https://istiod-basic-install.brice-project-istio.svc:443/inject?timeout=10s": no endpoints available for service "istiod-basic-install", reason=null, additionalProperties={})], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Internal error occurred: failed calling webhook "sidecar-injector.istio.io": failed to call webhook: Post "https://istiod-basic-install.gre05-br...
12719at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
12720at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:536) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
12721at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
12722at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:554) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]

Am I missing a StrimziPodSet parameter to ensure Scheduling and pod management?

[scholzj]

The error suggests the problem is with your Istio webhook and due to it Kubernetes cannot create the Pod as requested by Strimzi:

Internal error occurred: failed calling webhook "sidecar-injector.istio.io": failed to call webhook: Post "https://istiod-basic-install.brice-project-istio.svc:443/inject?timeout=10s": no endpoints available for service "istiod-basic-install"

So I guess you should remove the Istio admission controller or fix it to work. It is not clear what your intention is - Strimzi does not support Istio and its sidecars - although I think some people made it to work.

It is also not obvious why would the StatefulSets pods not be blocked by the same error, but it is hard to say without knowing your environment.

[Rabzou]

I was looking for a way to leave kafka outside istio, and I found that:
#817

It's already possile to add annotations to bypass istio.

[scholzj]

You can add annotations, sure. But I think the annotations will be interpreted by the Istio Admission controller. In the error you had at the beginning, the Istio Admission Controller did not seem to work. So I'm not sure the annotation will help.

[Rabzou]

During the drain, some pods are removed from a cluster and rescale on another.
So I guess the istiod pod is also moving... maybe during his move, StrimziPodSet can't reach him. that's why we get the error of the begining. So it stop, and don't scale kafka anymore.

Maybe in StatefulSet mode, there is much more retry operation, or a greater timeout. So when the istiod come back it can scale kafka pods. That's why we don't have error in this mode.

What do you think about this assumption ?

[scholzj]

That would be possible -> this would be a known problem with admission controllers. You should normally either have it HA or have it configured to let Kubernetes ignore it if it is not available. #8464 should add resync (periodical retry) in the next Strimzi version. But I think fixing the Istio controller should be done regardless of that.

[Rabzou]

Tested with strimzi 0.35.1, and there is no problem anymore.
Thanks @scholzj !