Strimzi
[Bug]: KubernetesClientException in ZooKeeper StrimziPodSet #8533
brsolomon-deloitte
started this conversation in
General
Replies: 7 comments 10 replies
-
|
I think you would need to share the full log. This looks like a problem with your Kubernetes environment - e.g. due to some misbehaving admission controllers etc. But hard to say without the full log. |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Which log(s) would be useful for me to share? |
Beta Was this translation helpful? Give feedback.
-
|
The cluster Operator log. |
Beta Was this translation helpful? Give feedback.
-
|
Possibly related to:
Regarding potential issues with the k8s environment - we have 10+ other applications deployed and haven't seen any issues with k8s itself thus far. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I do not think #8516 or #8528 are related here. #8514 is I think related. |
Beta Was this translation helpful? Give feedback.
-
|
Logs from |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
So, as you can see in the log: The operator is trying to create the pod and sends the requests to your Kubernetes API. But that request is not handled. As discussed in #8514 => this could be caused by some AdmissionWebhook which does not process the request (on time). So you need to look in that direction to find out more about the problem. |
Beta Was this translation helpful? Give feedback.
-
@scholzj thanks for your response - do you have any tips on how to debug further and where to look? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I do not know. I guess the first point would be to go through the webhooks and check which one is watching for Pods - that should eliminate some of them. |
Beta Was this translation helpful? Give feedback.
-
|
Found it. We have verified this timeout is directly caused by the Vertical Pod Autoscaler application's With the With the $ kubectl get mutatingwebhookconfiguration.admissionregistration.k8s.io/vpa-webhook-config -o yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
creationTimestamp: "2023-05-22T18:08:33Z"
generation: 1
name: vpa-webhook-config
resourceVersion: "6131859"
uid: 77208c43-5c2e-426d-b94a-eaa935b1a34b
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJU...
service:
name: vpa-webhook
namespace: kube-system
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: vpa.k8s.io
namespaceSelector: {}
objectSelector: {}
reinvocationPolicy: Never
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
- apiGroups:
- autoscaling.k8s.io
apiVersions:
- '*'
operations:
- CREATE
- UPDATE
resources:
- verticalpodautoscalers
scope: '*'
sideEffects: None
timeoutSeconds: 30 |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Interesting. Do you have any logs from VPA about what the issue on its side was? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately no. I thought that the VPA admission controller would show something, but it does not. (See kubernetes/autoscaler#5792 (comment).) |
Beta Was this translation helpful? Give feedback.
-
|
The webhook has timeout of 30 seconds. The Kubernetes Client has only 10 by default I think. That is something you can reconfigure -> either increase it in Strimzi or decrease it in the VPA webhook. |
Beta Was this translation helpful? Give feedback.
-
|
I have adjusted Where is the timeout configuration in the Strimzi Helm chart? |
Beta Was this translation helpful? Give feedback.
-
|
Its part of the client configuration: https://github.com/fabric8io/kubernetes-client#configuring-the-client => |
Beta Was this translation helpful? Give feedback.
-
|
I have opened kubernetes/autoscaler#5792 with the VPA app in question here. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Setting the following for the strimzi helm chart seems to be working (Pods now coming up): logLevel: DEBUG
operationTimeoutMs: 600000
extraEnvs:
- name: KUBERNETES_REQUEST_TIMEOUT
value: '60000'plus revert VPA to its standard Frustratingly I cannot get VPA to actually log what it is mad about, at any |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Bug Description
Bug Description
On the initial installation of Kafka using strimzi-kafka-operator, we are seeing an
HttpTimeoutException: request timed outin thedeployment/strimzi-kafka-operatorlogs and aReason=KubernetesClientExceptionon the ZooKeeperStrimziPodSet.Steps to reproduce
kafka-systemnamespace with followingvalues.yaml:elasticnamespace:Expected behavior
Kafka cluster comes up cleanly, or strimzi logs give more of an indication of what specifically is failing.
Strimzi version
0.35.0
Kubernetes version
1.26
Installation method
Helm
Infrastructure
Amazon EKS
Configuration files and logs
Additional context
We have all TCP traffic allowed between EKS worker nodes.
Beta Was this translation helpful? Give feedback.
All reactions