drain-cleaner not invoked when nodes are drained.

[raj-saxena]

We installed drain-cleaner recently to smoothen the process of upgrading node-pools using the helm-chart.

We are using GKE and our cluster has cert-manager so we went with default configuration and the only values that we had to change were:

namespace:
  create: false
  name: strimzi

I have verified that there is a ValidatingWebhookConfiguration -

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: strimzi/strimzi-drain-cleaner
    meta.helm.sh/release-name: strimzi-drain-cleaner
    meta.helm.sh/release-namespace: strimzi
  creationTimestamp: "2023-06-09T06:16:44Z"
  generation: 4
  labels:
    app: strimzi-drain-cleaner
    app.kubernetes.io/instance: strimzi-drain-cleaner
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: strimzi-drain-cleaner
    app.kubernetes.io/version: 0.4.2
    helm.sh/chart: strimzi-drain-cleaner-0.4.2
  name: strimzi-drain-cleaner
  resourceVersion: "829436629"
  uid: efe30ac2-ad9c-4d32-bfff-8f2b10ec1f2a
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0t....0tLQo=
    service:
      name: strimzi-drain-cleaner
      namespace: strimzi
      path: /drainer
      port: 443
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: strimzi-drain-cleaner.strimzi.io
  namespaceSelector: {}
  objectSelector: {}
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods/eviction
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 5

However, we drain the nodes, we don't see any logs as this blog says

Any suggestions to investigate?

[scholzj]

How do you drain the nodes? With kubectl drain as done in the blog posts? Or with some special GKE tooling? In general, the Kubernetes logs should have any logs if there are some issues. The Drain Cleaner itself is just an HTTP server which is receiving the events form Kubernetes. So there might be two issues -> does it emit the eviction events? And if it does, why are they not sent to the webhook.

[raj-saxena]

Yes, I used the kubectl drain command to train the nodes.
I checked the GKE audit logs and noticed the following logs -

What's weird is that the drainer didn't report receiving any requests.

I also see that the log's response part says there were too many requests as the reason.

Could it be because of some noise due to the eviction of other pods?

I was trying to drain multiple nodes at the same time. So, that activity is also a suspect.
I'll try reproducing the issue with just one node and update here.

[scholzj]

Hmm, weird. But if it failed, the Kubernetes logs should have more details. I think the controller manager might be the right component.

[dlouvier]

Hi @scholzj

Raj & me we work for the same team and I had some time today to look at this.

I believe it is a problem with GKE Firewall rules between CP <> AdmissionWebhook Listener Service, nothing related with Strimzi.

Endup with this conclusion because I configured another AdmissionWebhook in parallel to drain-cleaner and I also was not able to see any incoming request. Increased the timeout and it also timeout with a higher value, so it should be sort-kind of firewall aorund for what I've seen in other issues.

[dlouvier]

TL;DR: Solution for GKE at the end of the message

Hi again @scholzj

The main problem is here:

            - containerPort: 8443
              name: https

The drain-cleaner container is using a non-standard port and by default it seems to be blocked. I looked to another Admission Control implementations (Prometheus Operator and Cert Manager) they expose the port 10250 (kubelet).

❯ k get svc strimzi-drain-cleaner
NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
strimzi-drain-cleaner   ClusterIP   10.10.10.100   <none>        443/TCP   10d
❯ k get endpoints cert-manager-webhook
NAME                   ENDPOINTS           AGE
cert-manager-webhook   10.05.10.92:10250   11d

❯ k get pod cert-manager-webhook-5cd7ff21d9-hdkjl -o yaml
(...)
    - containerPort: 10250
      name: https
      protocol: TCP
(...)

---

• I know from other strimzi components, ports are usually with an 8.
• It seems it is not possible to configure the port for drain-cleaner

I think it could be worth for other users as well if drain-cleaner will run on port 10250 or at least if this could be configurable, so it is not necessary to add additional firewall rules to default GKE cluster (I bet could happen the same in other clouds)

Best regards,

---

Solution for GKE Clusters
(docs)

❯ gcloud compute firewall-rules create RULE_NAME \
    --action ALLOW \
    --direction INGRESS \
    --source-ranges CIDR_CONTROL_PLANE_MASTER \
    --rules tcp:8443 \
    --target-tags DST_TAGS

[scholzj]

TBH, I think that port 443 is the standard one for HTTPS communication 🤷 - not 10250 which is just a Kubelet port. And I don't think we can change the default as if others follow a similar logic, it might break it for someone else. That said, I think all that matters is the service port here which is where the control plane should connect (not 8443 in the container). So I think all you should need is to simply change the service and the webhook configuration to any port you want.

[raj-saxena]

I think all that matters is the service port here which is where the control plane should connect (not 8443 in the container)

Yeah, this was our assumption too but we had to upgrade the cluster again recently and I can confirm that we needed to add a firewall rule for allowing traffic from GKE master range to the nodes for the port 8443.
We believe that this is because the Service is a collection of network Endpoints that direct traffic to port 8443.

(These rules are needed for any communication. So, we have a rule for cert-manager too)

[mike-serchenia]

We have the same issue, the problem is specifically with node pool upgrade and not kubectl drain

during node pool upgrade gke doesn't send "operation":"CREATE","kind":"policy/v1, Kind=Eviction" if kafka spec has maxUnavailable eq 0 (PDB minAvailable: equal to number of kafka replicas)