How can we access kafka deployed by strimzi-operator from Pod that has istio-proxy sidecar?

[LittleWat]

Hello! Let me ask a question for the first time.

This is the sample repo I created:
https://github.com/LittleWat/strimzi-access-from-istio-test

I am deploying the kafka pod using strimzi-operator without istio-injection.

I want to control the ACL creation/deletion, so I disabled entity-operator.
Generating CertificateAuthority is also disabled for easy local development.
So the kafka deployment file is as follows:
https://github.com/LittleWat/strimzi-access-from-istio-test/blob/main/k8s/kafka-my-cluster.yaml

I want to access this Kafka pod from the pod that has istio-proxy sidecar.
But I cannot access it with the following errors:

$ kubectl logs -n kafka my-cluster-kafka-0
...
2023-02-23 12:13:34,442 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.244.0.23 (channelId=10.244.0.17:9093-10.244.0.23:38710-39) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SSL-7]
2023-02-23 12:13:35,406 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.244.0.23 (channelId=10.244.0.17:9093-10.244.0.23:38722-39) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(TLS-9093)-SSL-8]

This is Kiali UI.

My procedure is written in the readme.
https://github.com/LittleWat/strimzi-access-from-istio-test

How can I solve this? Any ideas are really appreciated 🙏 Thank you in advance!

[sreejesh-radhakrishnan-db]

Not sure about your setup. but in GKE in our case we use ISTIO Sidecar on all our application. except for strimzi namespace.
We dont do anything special, all we do is our connection string to kakfak is my-clusterbootstrap..svc and connection to kafka worked okay for us.

Seeing your error, i am assuming you are enabling MTLS for your connection and kafka do not trust your cert.

this is more like SSL handshake than connection due to istio being present.

[LittleWat]

Thank you for your quick reply!
Oh, you can make this work! It's nice to know that.

Seeing your error, i am assuming you are enabling MTLS for your connection and kafka do not trust your cert.

Yes, exactly.

this is more like SSL handshake than connection due to istio being present.
Should I investigate the certificate setup more?

I would like to know the difference between yours and mine.
I am using the local minikube. This might be a cause.

Sorry, but let me ask a question 🙇
Could you see any difference? Are you using the mTLS resources like this?
https://github.com/LittleWat/strimzi-access-from-istio-test/blob/main/k8s/istio-resources-for-app.yaml

[LittleWat]

I checked the access from the istio-proxy to the kafka pod with the following openssl command:

istio-proxy@kafka-cli-7dc6f888b5-747pv:/$ openssl s_client -connect my-cluster-kafka-bootstrap.kafka.svc:9093 -tls1_2 -CAfile /etc/certs/ca.crt -cert /etc/certs/tls.crt -key /etc/certs/tls.key
CONNECTED(00000003)
depth=1 C = JP, CN = test-ca.local
verify return:1
depth=0 CN = my-cluster-kafka-brokers
verify return:1
---
Certificate chain
 0 s:CN = my-cluster-kafka-brokers
   i:C = JP, CN = test-ca.local
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb 25 08:11:13 2023 GMT; NotAfter: Feb 22 08:11:13 2033 GMT
---
Server certificate

...

subject=CN = my-cluster-kafka-brokers
issuer=C = JP, CN = test-ca.local
---
Acceptable client certificate CA names
C = JP, CN = test-ca.local
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4406 bytes and written 2278 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: DB7E19200407513A43666450AE9A1C904AE2A1F9D1D6C07638F1A9738F3FCFB5
    Session-ID-ctx:
    Master-Key: 9518EE96147DADA274D7584C0827D138DDF7D44A4A046B53C4FF4B424702843716EAD13C5F266852A510B7E9E7ACE828
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    
     ...
    
    Start Time: 1677312818
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

This looks okay, so the SSL handshake might work well.
My istio configuration might have an issue. 🤔