Readiness,Startup,Liveness probe error: Strimzi Cluster Operator

[berbarroso]

Hi,

I'm having a problem with the Strimzi operator. I ran it on Minikube.

The Readiness, Startup, and Liveness probe of the pods connected to it failed.

strimzi-cluster-operator pod events

I followed this documentation very precisely and also tried to change the latest image versions.
https://debezium.io/documentation/reference/stable/operations/kubernetes.html

These are the log files

Some of the log errors on strimzi-cluster-operator pod
WARN PlatformFeaturesAvailability:156 - API Group route.openshift.io is not supported
WARN PlatformFeaturesAvailability:156 - API Group build.openshift.io is not supported
WARN PlatformFeaturesAvailability:156 - API Group image.openshift.io is not supported

WARN VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
WARN VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
WARN VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
WARN VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'

WARN VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
WARN VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
WARN VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'
WARN VersionUsageUtils:60 - The client is using resource type 'kafkaconnectors' with unstable version 'v1beta2'

ERROR AbstractOperator:284 - Reconciliation #2(watch) KafkaConnect(debezium-example/debezium-connect-cluster): createOrUpdate failed
io.strimzi.operator.common.model.InvalidConfigParameterException: database.password: A null value is not allowed for this key

AbstractOperator:547 - Reconciliation #11(timer) KafkaConnect(debezium-example/debezium-connect-cluster): Failed to reconcile database.password: A null value is not allowed for this key

Logs from other pods
Logs of debezium-connect-cluster-connect-0.pdf
Logs of debezium-cluster-entity-operator pod.pdf
Logs of debezium-cluster-kafka-0 pod.pdf
Logs of debezium-cluster-zookeeper-0 pod.pdf

What are the possible issues that I need to do?

Thank you in advance for answering. :)

[scholzj]

I guess you should make sure that the pods (and your cluster) have sufficient resources available. Insufficient resources - especially CPU can cause problems like this. The error from the Cluster Operator suggests you misconfigured something, but without a full log (preferably as text and not PDF) and the configuration, it is hard to say what exactly is the issue.

[berbarroso]

Noted on this, will try to check my resources and send a text file for logs. Thank you!

[berbarroso]

I tried to recreate my Minikube and set it to my maximum resources (CPUs=4, Memory=7886MB)

And it still have Readiness,Startup,Liveness probe error

These are the log files
strimzi-cluster-operator.log

debezium-connect-cluster-connect-0.log
debezium-cluster-zookeeper-0.log
debezium-cluster-kafka-0.log
debezium-cluster-entity-operator-5f796c8885-btn4b.log

[berbarroso]

When I try to include the username and password on creating a Debezium Connector yaml instead of connecting the secret (debezium-secret), this is the log from strimzi-cluster-operator pod.

INFO ClusterOperator:142 - Triggering periodic reconciliation for namespace *
INFO AbstractOperator:265 - Reconciliation #183(timer) Kafka(debezium-example/debezium-cluster): Kafka debezium-cluster will be checked for creation or modification
INFO AbstractOperator:265 - Reconciliation #184(timer) KafkaConnect(debezium-example/debezium-connect-cluster): KafkaConnect debezium-connect-cluster will be checked for creation or modification
INFO AbstractConnectOperator:440 - Reconciliation #184(timer) KafkaConnect(debezium-example/debezium-connect-cluster): creating/updating connector: debezium-connector-mysql
WARN AbstractConfiguration:147 - Reconciliation #184(timer) KafkaConnect(debezium-example/debezium-connect-cluster): Configuration option "tasks.max" is forbidden and will be ignored
WARN AbstractConnectOperator:917 - Reconciliation #184(timer) KafkaConnect(debezium-example/debezium-connect-cluster): Error reconciling connector debezium-connector-mysql
io.strimzi.operator.cluster.operator.assembly.ConnectRestException: PUT /connectors/debezium-connector-mysql/config returned 400 (Bad Request): Connector configuration is invalid and contains the following 1 error(s):
Unable to connect: Access denied for user 'ZGViZXppdW0='@'10.244.0.19' (using password: YES)

This is the whole log file.
strimzi-cluster-operator 2.log

This is my configuration for Debezium Connector

_cat << EOF | kubectl create -n debezium-example -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: debezium-connector-mysql
labels:
strimzi.io/cluster: debezium-connect-cluster
spec:
class: io.debezium.connector.mysql.MySqlConnector
tasksMax: 1
config:
tasks.max: 1
database.hostname: mysql
database.port: 3306
database.user: ZGViZXppdW0= # instead of "${secrets:debezium-example/debezium-secret:username}"
database.password: ZGJ6 #instead of "${secrets:debezium-example/debezium-secret:password}"
database.server.id: 184054
topic.prefix: mysql
database.include.list: inventory
schema.history.internal.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092
schema.history.internal.kafka.topic: schema-changes.inventory
EOF

[scholzj]

I'm afraid I do not know why the probes are failing. From the logs, things seem to work. So not sure why would they not pass the healthchecks.

The error from the connector seems to be a bit easier to understand. The connector is telling you that your username/password are wrong or do not have the rights. I wonder if you really should pass the values as base64 encoded. When you don't load them from Kubernetes Secrets, you should put them there in plaintext I guess.