Kafka zookeeper keeps on crashing

[irisityerikwaenerlund]

Hi,
My nodes in the zookeeper cluster keeps on crashing.
One of the secondary nodes suddenly krashes with:
2022-03-23T10:14:20.370962242Z
2022-03-23 10:14:20,370 WARN Exception when following the leader (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=0.0.0.0:2181)]

Then I get an javaEOF exeption.

My kafka config:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: devops
spec:
kafka:
replicas: 3
version: 2.8.1
resources:
requests:
memory: 4Gi
cpu: "1"
limits:
memory: 8Gi
cpu: "2"
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
jvmOptions:
-Xms: 512m
-Xmx: 512m
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
config:
auto.create.topics.enable: "true"
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: 2.8.1
max.message.bytes: 5000000
socket.recieve.bytes: 5000000
socket.send.buffer.bytes: 5000000
replica.fetch.max.bytes: 5000000
ssl.cipher.suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
ssl.enabled.protocols: "TLSv1.2"
ssl.protocol: "TLSv1.2"
storage:
type: ephemeral
zookeeper:
replicas: 3
resources:
requests:
memory: 1Gi
cpu: "1"
limits:
memory: 1Gi
cpu: "1"
jvmOptions:
-Xms: 512m
-Xmx: 512m
storage:
type: ephemeral

After the crash i get this:
WARN Cannot open channel to 2 at election address devops-zookeeper-1.cluster-zookeeper-nodes.kafka.svc/10.28.2.20:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-3]

[irisityerikwaenerlund]

I tried this in GKE autopilot and saw others having issues with this. Will test using GKE standard and see if this works.

[moritz89]

This still seems to be a problem (periodic SIGKILL after 5-20 minutes). Zookeeper and Kafka run stably on standard GKE clusters.

[sreejesh123]

I am getting same issue , what is fix? What does a standard GKE cluster means , please can you share the GKE settings used ?

[moritz89]

When creating a new cluster choose standard instead of autopilot: https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters

[sreejesh123]

Is there any fix planned to make strimzi work with auto scaling ?

[sreejesh123]

*auto pilot