Liveliness and Readiness Probe failing

[Utkarsh-0211]

Hi All, each time I'm deploying my cluster.yaml my kakfa and zookeeper pods are going into crashloop backoff due to Liveliness and Readiness Probe failing

strimzi version: latest
kafka-cluster.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.1.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.1"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 500Mi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 500Mi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

kubectl describe po my-cluster-entity-operator-67cbb8f86f-g4bq7 -n kafka

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  25m                   default-scheduler  Successfully assigned kafka/my-cluster-entity-operator-67cbb8f86f-g4bq7 to minikube
  Normal   Created    25m                   kubelet            Created container topic-operator
  Normal   Started    25m                   kubelet            Started container user-operator
  Normal   Started    25m                   kubelet            Started container topic-operator
  Normal   Pulled     25m                   kubelet            Container image "quay.io/strimzi/operator:0.28.0" already present on machine
  Normal   Created    25m                   kubelet            Created container user-operator
  Normal   Pulled     25m                   kubelet            Container image "quay.io/strimzi/kafka:0.28.0-kafka-3.1.0" already present on machine
  Normal   Created    25m                   kubelet            Created container tls-sidecar
  Normal   Started    25m                   kubelet            Started container tls-sidecar
  Warning  Unhealthy  25m                   kubelet            Liveness probe failed: Get "http://172.17.0.6:8081/healthy": dial tcp 172.17.0.6:8081: connect: connection refused
  Warning  Unhealthy  25m (x2 over 25m)     kubelet            Readiness probe failed: Get "http://172.17.0.6:8081/ready": dial tcp 172.17.0.6:8081: connect: connection refused
  Normal   Killing    23m                   kubelet            Container topic-operator failed startup probe, will be restarted
  Warning  Unhealthy  5m48s (x96 over 25m)  kubelet            Startup probe failed: Get "http://172.17.0.6:8080/healthy": dial tcp 172.17.0.6:8080: connect: connection refused
  Normal   Pulled     47s (x9 over 25m)     kubelet            Container image "quay.io/strimzi/operator:0.28.0" already present on machine

[Utkarsh-0211]

@scholzj Can you please look into and help where the issue might be in this please.

[scholzj]

Sorry, but it is not really clear what is actually failing and what is happening from what you provided. Looking at the YAML, I see several possible issues:

• 500Mi storage is not enough. I normally start with 100Gi, but I think you should have 10Gi as a bare minimum if you want any stability
• There are no resources configurations, so it is not clear whether it has enough resources. You should make sure:
  • The cluster has enough resources
  • There are no LimitRange resources on your cluster whcih would give it some super-small defaults
  • Ideally configure some reasonable resources for each component

[f3r73ch]

In my case, it was resolved by increasing the corresponding times for the readiness (and also liveness) probes.

Also a similar reference here

cheers