Failed to connect to Zookeeper kafka-cluster-zookeeper

[chanvoyage]

hello everyone

1 ). i try to use TLS client authentication follow this blog , everything look good when Connect kafka cluster via ingress i found error : "Java.security.cert.CertificateException: No subject alternative DNS name matching strimzi-kafka-"
note: I use ingress nginx
https://itnext.io/kafka-on-kubernetes-the-strimzi-way-part-3-19cfdfe86660

2. i found same issues : i don't know root case or not but i try to use KUBERNETES_SERVICE_DNS_DOMAIN
   no subject alternative DNS name matching strimzi-kafka-0.strimzi-kafka-brokers.strimzi.svc.cumulus-k8s found. #1486

3. add env KUBERNETES_SERVICE_DNS_DOMAIN to kafka operator and set useServiceDnsDomain: true to kafka cluster
   and then kubectl apply kafka-clueter.taml
   kafka operator logs tel me Failed to connect to Zookeeper kafka-cluster-zookeeper

cluster-oper.log

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka-cluster
namespace: aia-kafka
spec:
kafka:
version: 2.8.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
configuration:
useServiceDnsDomain: true
- name: tls
port: 9093
type: internal
tls: true
- name: external
port: 9094
type: ingress
tls: true
authentication:
type: tls
configuration:
bootstrap:
host: bootstrap.chaiyapon.com
brokers:
- broker: 0
host: broker-0.chaiyapon.com
- broker: 1
host: broker-1.chaiyapon.com
- broker: 2
host: broker-2.chaiyapon.com
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.8"
inter.broker.protocol.version: "2.8"
storage:
type: jbod
volumes:
- id: 0
size: 20Gi
type: persistent-claim
class: managed-premium
deleteClaim: true
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 20Gi
class: managed-premium
deleteClaim: true
entityOperator:
topicOperator: {}
userOperator: {}

=================

apiVersion: apps/v1
kind: Deployment
metadata:
name: strimzi-cluster-operator
labels:
app: strimzi
spec:
replicas: 1
selector:
matchLabels:
name: strimzi-cluster-operator
strimzi.io/kind: cluster-operator
template:
metadata:
labels:
name: strimzi-cluster-operator
strimzi.io/kind: cluster-operator
spec:
serviceAccountName: strimzi-cluster-operator
volumes:
- name: strimzi-tmp
emptyDir:
medium: Memory
- name: co-config-volume
configMap:
name: strimzi-cluster-operator
containers:
- name: strimzi-cluster-operator
image: acrRegistry/quay.io/strimzi/operator:0.23.0
ports:
- containerPort: 8080
name: http
args:
- /opt/strimzi/bin/cluster_operator_run.sh
volumeMounts:
- name: strimzi-tmp
mountPath: /tmp
- name: co-config-volume
mountPath: /opt/strimzi/custom-config/
env:
- name: STRIMZI_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBERNETES_SERVICE_DNS_DOMAIN
value: chaiyapon.com
- name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
value: "120000"
- name: STRIMZI_OPERATION_TIMEOUT_MS
value: "300000"
- name: STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE
value: acrRegistry/quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE
value: acrRegistry/quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE
value: acrRegistry/quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE
value: acrRegistry/quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_KAFKA_IMAGES
value: |
2.6.0=quay.io/strimzi/kafka:0.23.0-kafka-2.6.0
2.6.1=quay.io/strimzi/kafka:0.23.0-kafka-2.6.1
2.6.2=quay.io/strimzi/kafka:0.23.0-kafka-2.6.2
2.7.0=quay.io/strimzi/kafka:0.23.0-kafka-2.7.0
2.8.0=quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_KAFKA_CONNECT_IMAGES
value: |
2.6.0=quay.io/strimzi/kafka:0.23.0-kafka-2.6.0
2.6.1=quay.io/strimzi/kafka:0.23.0-kafka-2.6.1
2.6.2=quay.io/strimzi/kafka:0.23.0-kafka-2.6.2
2.7.0=quay.io/strimzi/kafka:0.23.0-kafka-2.7.0
2.8.0=quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_KAFKA_CONNECT_S2I_IMAGES
value: |
2.6.0=quay.io/strimzi/kafka:0.23.0-kafka-2.6.0
2.6.1=quay.io/strimzi/kafka:0.23.0-kafka-2.6.1
2.6.2=quay.io/strimzi/kafka:0.23.0-kafka-2.6.2
2.7.0=quay.io/strimzi/kafka:0.23.0-kafka-2.7.0
2.8.0=quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_KAFKA_MIRROR_MAKER_IMAGES
value: |
2.6.0=quay.io/strimzi/kafka:0.23.0-kafka-2.6.0
2.6.1=quay.io/strimzi/kafka:0.23.0-kafka-2.6.1
2.6.2=quay.io/strimzi/kafka:0.23.0-kafka-2.6.2
2.7.0=quay.io/strimzi/kafka:0.23.0-kafka-2.7.0
2.8.0=quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES
value: |
2.6.0=quay.io/strimzi/kafka:0.23.0-kafka-2.6.0
2.6.1=quay.io/strimzi/kafka:0.23.0-kafka-2.6.1
2.6.2=quay.io/strimzi/kafka:0.23.0-kafka-2.6.2
2.7.0=quay.io/strimzi/kafka:0.23.0-kafka-2.7.0
2.8.0=quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
- name: STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE
value: quay.io/strimzi/operator:0.23.0
- name: STRIMZI_DEFAULT_USER_OPERATOR_IMAGE
value: quay.io/strimzi/operator:0.23.0
- name: STRIMZI_DEFAULT_KAFKA_INIT_IMAGE
value: quay.io/strimzi/operator:0.23.0
- name: STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE
value: quay.io/strimzi/kafka-bridge:0.19.0
- name: STRIMZI_DEFAULT_JMXTRANS_IMAGE
value: quay.io/strimzi/jmxtrans:0.23.0
- name: STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE
value: quay.io/strimzi/kaniko-executor:0.23.0
- name: STRIMZI_OPERATOR_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: STRIMZI_FEATURE_GATES
value: ""
livenessProbe:
httpGet:
path: /healthy
port: http
initialDelaySeconds: 10
periodSeconds: 30
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 10
periodSeconds: 30
resources:
limits:
cpu: 1000m
memory: 384Mi
requests:
cpu: 200m
memory: 384Mi
strategy:
type: Recreate

[scholzj]

• Formatting the YAML snippets would make this much more readable. Right now, it is not clear how do your whitespaces look like and that is important for YAML.
• If the error happened in Connect, you should probably share the KafkaConnect YAML and the the log with the error.
• KUBERNETES_SERVICE_DNS_DOMAIN is the service domain of your Kubernetes cluster. Is chaiyapon.com really the service domain of your Kubernetes cluster? You use the same domain in the Ingress configuration which has nothing to do with the Kubernetes service domain. Using the same domain for Ingress and for the service domain would be a bit weird I think. It would cause all kind of DNS issues probably. So please double check in your Kubernetes configuration if this is really the case.

[chanvoyage]

cert.txt

[scholzj]

/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate suggests that your Ingres doesn't do TLS passthrough. Otherwise the client would get the Strimzi certificates. So you need to check the configuration if TLs passthrough is enabled.

[chanvoyage]

i doesn't enable tls passthrough on nginx controller thank you , but i run kafak-consumer again logs tel me
SSL handshake failed

i follow step https://itnext.io/kafka-on-kubernetes-the-strimzi-way-part-2-43192f1dd831

[root@jumphost user]# cat server-authen
bootstrap.servers=bootstrap.chaiyapon.com:443
security.protocol=SSL
ssl.truststore.location=/opt/jdk/jdk1.8.0_60/jre/lib/security/cacerts
//for JDK truststore, the default password is "changeit"
ssl.truststore.password=changeit

===================
openssl.txt

[root@jumphost kafka_2.12-2.8.0]# bin/kafka-console-consumer.sh --bootstrap-server bootstrap.chaiyapon.com:443 --topic chanza-topic --consumer.config /root/user/server-authen --from-beginning
[2021-07-02 20:37:54,714] WARN The configuration '//for' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2021-07-02 20:37:55,786] ERROR [Consumer clientId=consumer-console-consumer-93556-1, groupId=console-consumer-93556] Connection to node -1 (bootstrap.chaiyapon.com/20.198.151.244:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2021-07-02 20:37:55,787] WARN [Consumer clientId=consumer-console-consumer-93556-1, groupId=console-consumer-93556] Bootstrap broker bootstrap.chaiyapon.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-07-02 20:37:55,795] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1440)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:509)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:245)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1261)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:443)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:102)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:76)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:53)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:196)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
... 19 more
Processed a total of 0 messages

[scholzj]

Everything is very hard to read when you don't format it properly. But I think you configured TLS authentication. So you need to create a user and use keystore in the client.

[chanvoyage]

it work !!! thank you very much scholzi