Strimzi
Zookeeper pods keep restarting every 60s #7334
morningspace
started this conversation in
General
Replies: 2 comments 4 replies
-
|
You should start by sharing your configuration and all your logs. Without that it is hard to say what is it doing and why. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@scholzj Thanks for your prompt reply! Which configuration and logs are you referring to? I thought for the zookeeper configuration, it has been included in the above logs, or you want to check the full logs of the zookeeper pod? I pasted the full logs as below. Hope that helps. |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
The |
Beta Was this translation helpful? Give feedback.
-
|
@scholzj Since I've stopped the operator (scale replica down to 0), just want to filter out noise and narrow down the issue, I thought the logs for the operator and the But if you think that's needed, I can share them all here. Just that, once I relaunch the operator, everything that I made against the kafka and zookeeper pods manually will be reset, e.g.: the liveness and readiness probes will go back, the number of zookeeper and kafka pods will go back to 3. :-) |
Beta Was this translation helpful? Give feedback.
-
|
FYI: Below is the apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"kafka.strimzi.io/v1beta2","kind":"Kafka","metadata":{"annotations":{},"name":"my","namespace":"my-kafka"},"spec":{"kafka":{"config":{"offsets.topic.replication.factor":3,"transaction.state.log.min.isr":3,"transaction.state.log.replication.factor":3},"listeners":[{"name":"plain","port":9092,"tls":false,"type":"internal"}],"replicas":3,"storage":{"type":"jbod","volumes":[{"deleteClaim":true,"id":0,"size":"50Gi","type":"persistent-claim"}]},"version":"3.1.0"},"zookeeper":{"replicas":3,"storage":{"deleteClaim":true,"size":"10Gi","type":"persistent-claim"}}}}
creationTimestamp: "2022-08-22T06:32:56Z"
generation: 1
managedFields:
- apiVersion: kafka.strimzi.io/v1beta2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:kafka:
.: {}
f:config:
.: {}
f:offsets.topic.replication.factor: {}
f:transaction.state.log.min.isr: {}
f:transaction.state.log.replication.factor: {}
f:listeners: {}
f:replicas: {}
f:storage:
.: {}
f:type: {}
f:volumes: {}
f:version: {}
f:zookeeper:
.: {}
f:replicas: {}
f:storage:
.: {}
f:deleteClaim: {}
f:size: {}
f:type: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-08-22T06:32:56Z"
- apiVersion: kafka.strimzi.io/v1beta2
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:clusterId: {}
f:conditions: {}
f:listeners: {}
f:observedGeneration: {}
manager: okhttp
operation: Update
time: "2022-09-15T04:44:01Z"
name: my
namespace: my-kafka
resourceVersion: "26675903"
uid: 98719e78-17a8-47f1-a631-7f224d9b9cb2
spec:
kafka:
config:
offsets.topic.replication.factor: 3
transaction.state.log.min.isr: 3
transaction.state.log.replication.factor: 3
listeners:
- name: plain
port: 9092
tls: false
type: internal
replicas: 3
storage:
type: jbod
volumes:
- deleteClaim: true
id: 0
size: 50Gi
type: persistent-claim
version: 3.1.0
zookeeper:
replicas: 3
storage:
deleteClaim: true
size: 10Gi
type: persistent-claim
status:
clusterId: Spfxxs89Q46wAC7ib_qsJg
conditions:
- lastTransitionTime: "2022-09-15T04:40:06.923442Z"
message: default.replication.factor option is not configured. It defaults to 1
which does not guarantee reliability and availability. You should configure
this option in .spec.kafka.config.
reason: KafkaDefaultReplicationFactor
status: "True"
type: Warning
- lastTransitionTime: "2022-09-15T04:40:06.924515Z"
message: min.insync.replicas option is not configured. It defaults to 1 which
does not guarantee reliability and availability. You should configure this option
in .spec.kafka.config.
reason: KafkaMinInsyncReplicas
status: "True"
type: Warning
- lastTransitionTime: "2022-09-15T04:44:01.903Z"
status: "True"
type: Ready
listeners:
- addresses:
- host: my-kafka-bootstrap.my-kafka.svc
port: 9092
bootstrapServers: my-kafka-bootstrap.my-kafka.svc:9092
name: plain
type: plain
observedGeneration: 1 |
Beta Was this translation helpful? Give feedback.
-
|
Well, that is where the operator log comes in which I asked about before ;-). We need to figure out what is restarting it. Does it shutdown it self? Does the operator restart it? Do the healt hchecks kill it? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I deployed strimzi kafka on my env using the operator but found that kafka is failed to start as zookeeper keeps restarting. After I did the following things just for troubleshooting:
I can see there's only one zookeeper pod launched but still keeps restarting every 60s. When checking the pod logs, it seems everything looks good. The pod logs will stop at the last line, "Peer state changed: leading ... ", till pod restarted. Any idea why it happens?
Here is the log details:
Beta Was this translation helpful? Give feedback.
All reactions