Need help about installing my own cert and key #6867

akametall · 2022-05-25T15:30:08Z

akametall
May 25, 2022

try to do this https://strimzi.io/docs/operators/latest/full/configuring.html#kafka-listener-certificates-str
and getting tls handshake error
cert are selfsigned, generated by vault
question is

i must use client.key and client.crt in secret?
do i need to change ca in strimzi secrets?

scholzj · 2022-05-25T15:32:40Z

scholzj
May 25, 2022
Maintainer

Sorry, but I think you will need to:

Properly describe what you did
Share what exact errors are you getting with what configurations

5 replies

akametall May 25, 2022
Author

`listeners:

name: plain
port: 9092
type: internal
tls: false
name: tls
port: 9093
type: internal
tls: true
name: external
port: 9094
type: loadbalancer
tls: true
authentication:
type: tls
configuration:
brokerCertChainAndKey:
secretName: my-secret
certificate: kafka.crt
key: kafka.key
bootstrap:
loadBalancerIP: 0
brokers:
- broker: 0
  loadBalancerIP: 0

authorization:
type: simple`

akametall May 25, 2022
Author

openssl s_client -connect ***:9094 -servername ***

CONNECTED(00000003)

depth=0 CN = my-client.clients.kafka.mts.ru

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=0 CN = my-client.clients.kafka.mts.ru

verify error:num=21:unable to verify the first certificate

verify return:1

Certificate chain

0 s:CN = my-client.clients.kafka.mts.ru

i:CN = Acme Kafka Intermediate CA

Server certificate

-----END CERTIFICATE-----

subject=CN = my-client.clients.kafka.mts.ru

issuer=CN = Acme Kafka Intermediate CA

Acceptable client certificate CA names

O = io.strimzi, CN = clients-ca v0

Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA 384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1

Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA- PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512

Peer signing digest: SHA256

Peer signature type: RSA-PSS

Server Temp Key: X25519, 253 bits

SSL handshake has read 1556 bytes and written 413 bytes

Verification error: unable to verify the first certificate

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384

Server public key is 2048 bit

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

Early data was not sent

Verify return code: 21 (unable to verify the first certificate)

139622523446592:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:

scholzj May 25, 2022
Maintainer

It is not really readable when not properly formatted. But I'm not sure what is really the issue here. It seems the broker is using some custom certificates.

akametall May 25, 2022
Author

sorry, i try to copy it, but something went wrong.

issue is then i try to connect to kafka i get

from client

anivan54@test2:~$ kafkacat -b [10.73.96.58:9094](http://10.73.96.58:9094/) -F client.properties -t topic -C -e Reading configuration from file client.properties ERROR: Failed to query metadata for topic topic: Local: Broker transport failure

from kafka

2022-05-25 14:23:00,969 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /100.64.32.3 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-13]

scholzj May 25, 2022
Maintainer

I do not use kafkacat + you don't share anyway the configuration, so not sure how did you configured it. You really need to properly describe what you did -> which certificates you used where etc.

akametall · 2022-05-26T14:02:10Z

akametall
May 26, 2022
Author

First - i'm sorry for my english and stupidity
Second - try to write:

i generate in vault CA and intemediate CA (combine this to docs https://opencredo.com/blogs/securing-kafka-using-vault-pki/ https://strimzi.io/blog/2018/11/16/using-vault-with-strimzi/)

and now i have root.crt and kafka-int-ca.pem

create roles wit allowed_domains clients.kafka.acme.com and llowed_domains=servers.kafka.acme.com
configuring kafka server cert and key:
prp-kafka-server.key and prp-kafka-server.crt

i get it from pem with this commands
jq -r '.data.private_key' >
jq -r '.data.certificate' >

next i put prp-kafka-server.key and prp-kafka-server.crt into secret with this

kubectl create secret generic my-secret --from-file=prp-kafka-server.key --from-file=prp-kafka-server.crt

Commands i take from here https://strimzi.io/docs/operators/latest/full/configuring.html#kafka-listener-certificates-str

next i doing this wtih deployment
- name: external
  port: 9094
  type: loadbalancer
  tls: true
  authentication:
  type: tls
  configuration:
  brokerCertChainAndKey:
  secretName: my-secret
  certificate: prp-kafka-server.crt
  key: prp-kafka-server.key
  bootstrap:
  loadBalancerIP: lb-0000-s2-k8s-sstore-*
  brokers:
  - broker: 0
  loadBalancerIP: lb-0000-s2-k8s-sstore-*

and kubectl apply -f myyml.yml

Watching my kafka correctly starts
Next i generate client pem with something like

vault write -field certificate kafka-int-ca/issue/kafka-client
common_name=my-client.clients.kafka.acme.com format=pem_bundle > client.pem

doing somth like
openssl pkcs12 -inkey client.pem -in client.pem -name client -export -out client.p12

keytool -importkeystore -deststorepass smtpass
-destkeystore kafka-keystore.jks -srckeystore client.p12 -srcstoretype PKCS12

create some client.properties file like this

security.protocol=SSL
ssl.keystore.location=/home/kafka/kafka-keystore.jks
ssl.keystore.password=smthpass
ssl.key.password=smthpass
ssl.ca.location=/home/kafka/root.crt

*now i know that you didn't work with kafkacat but it seems like client.props are the same

next i trying this and get that

@test2:~$ kafkacat -b ***8:9094 -F client.properties -t topic -C -e
% Reading configuration from file client.properties
% ERROR: Failed to query metadata for topic topic: Local: Broker transport failure

And i think this is not the problem of kcat. Somewhere i made a mistake/s

maybe somebody try doing something like this....

2 replies

akametall May 26, 2022
Author

and from kafka side i get this

INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with / (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-13]

scholzj May 26, 2022
Maintainer

The listener certificate is only the server certificate for Kafka. (i.e. the keystore in the Kafka configuration). So on your client, it would correspond to the truststore / CA certificate. So IMHO the problem is in steps 7 and 8.

The client certificate has to be based on the Clients CA. You can generate it using the User Operator. Or you can provide your own Clients CA from Vault and then you can get the certificate like you do today.

Another things which seems weird is that you pass it the JKS keystore. I don't think kafkacat will be able to use that.

As I said, I do not use kafkacat, so I'm not sure the error corresponds to it. But this is something I see from your steps.

Maybe the easiest to start with would be to divide the concerns:

First disable the TLS authentication in the listener and remove the keystore configuration and doublecheck tht it works without the authentication.
Only then, once the first step works, enable the client authentication again and try to get that working.

akametall · 2022-05-26T14:38:50Z

akametall
May 26, 2022
Author

openssl give me smth like this

test2:$ openssl s_client -connect ...:9094 -servername ....
CONNECTED(00000003)
depth=0 CN = prp-kafka.servers.kafka
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = prp-kafka.servers.kafka
verify error:num=21:unable to verify the first certificate
verify return:1

Certificate chain
0 s:CN = prp-kafka.servers.kafka.mts.ru
i:CN = kafka intermediate ca

Server certificate
-----BEGIN CERTIFICATE-----
MIID+TCCAuGgAwIBAgIUGLce/Xq5u9rS9Zu7mI5GPPJMzmgwDQYJKoZIhvcNAQEL
......
Qqvxrna0FXZWLMh1FA==
-----END CERTIFICATE-----
subject=CN = prp-kafka.servers.kafka

issuer=CN = kafka intermediate ca

Acceptable client certificate CA names
O = io.strimzi, CN = clients-ca v0
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits

SSL handshake has read 1683 bytes and written 413 bytes
Verification error: unable to verify the first certificate

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)

140333897921856:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:....

5 replies

scholzj May 26, 2022
Maintainer

I think the outcome here is expected. It says that it:

Uses the listener certificate as server certificate
Uses the Strimzi Clients CA for the client authentication (see the point in the previous comment)

But because you did not provided the certificates, the OpenSSL will always fail.

akametall May 27, 2022
Author

So i can use the cert and pass that i generate with user-operator and just provide prp-kafka-server.crt as CA, am i right?

scholzj May 27, 2022
Maintainer

Yes, correct. That is the idea.

akametall May 27, 2022
Author

Thank you for your time and help.
Try to do this.

scholzj May 27, 2022
Maintainer

Give it a try and let me know. I will be traveling next week, so I might be slower to response. But we will get it working eventually.

akametall · 2022-05-29T07:49:29Z

akametall
May 29, 2022
Author

Does not work.
Try to write it again from start. Maybe you can find mistake:

Start from vault
1 vault secrets enable pki
2 vault secrets enable -path root-ca pki
3 vault secrets tune -max-lease-ttl=8760h root-ca
4. vault write -field=certificate root-ca/root/generate/internal common_name="kafka root ca" ttl=87600h > root.crt
5. vault write root-ca/config/urls issuing_certificates="http://127.0.0.1:8200/v1/root-ca/ca" crl_distribution_points="http://127.0.0.1:8200/v1/root-ca/crl"
6. vault secrets enable -path=kafka-int-ca pki
7. vault secrets tune -max-lease-ttl=43800h kafka-int-ca
8. vault write -format=json kafka-int-ca/intermediate/generate/exported common_name="kafka intermediate ca" ttl="43800h" format="pem" > kafka-int-ca
9. vault write -format=json root-ca/root/sign-intermediate [email protected] format=pem ttl=43800h > kafka-int-ca
10. vault write kafka-int-ca/intermediate/set-signed [email protected]
11. vault write kafka-int-ca/config/urls issuing_certificates="http://127.0.0.1:8200/v1/kafka-int-ca/ca" crl_distribution_points="http://127.0.0.1:8200/v1/kafka-int-ca/crl"
12. vault write kafka-int-ca/roles/kafka-server allowed_domains=servers.kafka.smth.com allow_subdomains=true max_ttl=72h
13. cat > kafka-server.hcl <<EOF
path "kafka-int-ca/issue/kafka-server" {
capabilities = ["update"]
}
EOF
14. vault policy write kafka-server kafka-server.hcl
15. vault write auth/token/roles/kafka-server allowed_policies=kafka-server period=24h
16. vault token create -role kafka-server
17. export VAULT_TOKEN=smthsmth
18. vault write -format=json kafka-int-ca/issue/kafka-server common_name=prp-kafka.servers.kafka.smth.com format=pem > prp-kafka-server

0 replies

akametall · 2022-05-29T07:54:25Z

akametall
May 29, 2022
Author

Next deploy kafka

kubectl create secret generic my-secret --from-file=prp-kafka-server.key --from-file=prp-kafka-server.crt
spec:
kafka:
version: 3.0.0
replicas: 1
listeners:
- name: plain
  port: 9092
  type: internal
  tls: false
- name: tls
  port: 9093
  type: internal
  tls: true
- name: external
  port: 9094
  type: loadbalancer
  tls: true
  authentication:
  type: tls
  configuration:
  brokerCertChainAndKey:
  secretName: my-secret
  certificate: prp-kafka-server.crt
  key: prp-kafka-server.key
  bootstrap:
  loadBalancerIP: fff
  brokers:
  - broker: 0
    loadBalancerIP: fff

- broker: 1

loadBalancerIP:

- broker: 2

loadBalancerIP: fff

authorization:
  type: simple

superUsers:

- CN=kafka-admin

kubectl apply -f kafka-prp-kafka3.0.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
name: kafka-admin
labels:
strimzi.io/cluster: prp-cluster
spec:
authentication:
type: tls
kubectl apply -f .\kafka-user-kafka-admin.yaml

0 replies

akametall · 2022-05-29T07:55:28Z

akametall
May 29, 2022
Author

kubectl --kubeconfig=.conf get secret kafka-admin -o jsonpath='{.data.user.password}' | base64 --decode > kafka-admin.password
kubectl --kubeconfig=.conf get secret kafka-admin -o jsonpath='{.data.user.p12}' | base64 --decode > kafka-admin.p12

0 replies

akametall · 2022-05-29T07:56:28Z

akametall
May 29, 2022
Author

keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file prp-kafka-server.crt
keytool -importkeystore -srckeystore kafka-admin.p12 -destkeystore kafka.client.keystore.jks -srcstoretype pkcs12

0 replies

akametall · 2022-05-29T07:57:51Z

akametall
May 29, 2022
Author

1.security.protocol=SSL
ssl.keystore.location=/home/fff/kafka.client.keystore.jks
ssl.keystore.password=changeme
ssl.key.password=fffffffff
ssl.truststore.location=/home/fff/kafka.client.truststore.jks
ssl.truststore.password=changeme

2../bin/kafka-console-producer.sh --bootstrap-server 10.73.96.58:9094 -topic mytopic --producer.config ./bin/client.properties

0 replies

akametall · 2022-05-29T07:58:34Z

akametall
May 29, 2022
Author

and finally get this

org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:439)
at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:290)
at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:317)
at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:302)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:45)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafka.common.KafkaException: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createSSLContext(DefaultSslEngineFactory.java:268)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:173)
at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140)
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97)
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:447)
at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:428)
... 5 more
Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is us ed during decryption.
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:446)
at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:90)
at java.base/java.security.KeyStore.getKey(KeyStore.java:1057)
at java.base/sun.security.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:145)
at java.base/sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70)
at java.base/javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:271)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createSSLContext(DefaultSslEngineFactory.java:251)
... 14 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975)
at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1056)
at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
at java.base/com.sun.crypto.provider.PBES2Core.engineDoFinal(PBES2Core.java:323)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:387)
at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:276)
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:381)
... 20 more

0 replies

akametall · 2022-05-29T08:29:08Z

akametall
May 29, 2022
Author

self answer

if here

1.security.protocol=SSL
ssl.keystore.location=/home/fff/kafka.client.keystore.jks
ssl.keystore.password=changeme
ssl.key.password=fffffffff
ssl.truststore.location=/home/fff/kafka.client.truststore.jks
ssl.truststore.password=changeme

i change ssl.key.password to changeme i start get from client this
[2022-05-29 11:28:28,551] WARN [Producer clientId=console-producer] Bootstrap broker 10.73.96.58:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-05-29 11:28:29,780] ERROR [Producer clientId=console-producer] Connection to node -1 (/10.73.96.58:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-05-29 11:28:29,780] WARN [Producer clientId=console-producer] Bootstrap broker 10.73.96.58:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

from kafka it looks like
2022-05-29 08:27:10,296 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /100.64.32.3 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-13]

0 replies

akametall · 2022-05-29T08:34:55Z

akametall
May 29, 2022
Author

And selfanswering again

added in client.properties tthis

ssl.endpoint.identification.algorithm:

get from kafka this:

2022-05-29 08:28:53,161 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]
2022-05-29 08:28:53,161 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
2022-05-29 08:32:08,991 INFO Principal = User:CN=kafka-admin is Denied Operation = Write from host = 100.64.32.3 on resource = Topic:UNKNOWN:NONE for request = InitProducerId with resourceRefCount = 0 (kafka.authorizer.logger) [data-plane-kafka-request-handler-3]

and from client this

:~/kafka$ ./bin/kafka-console-producer.sh --bootstrap-server 10.73.96.58:9094 -topic mytopic --producer.config ./bin/client.properties

asdf
[2022-05-29 11:32:52,813] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {mytopic=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2022-05-29 11:32:52,813] ERROR [Producer clientId=console-producer] Topic authorization failed for topics [mytopic] (org.apache.kafka.clients.Metadata)
[2022-05-29 11:32:52,815] ERROR Error when sending message to topic mytopic with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [mytopic]

0 replies

akametall · 2022-05-31T11:10:12Z

akametall
May 31, 2022
Author

It works with server crt and key and client's by strimzi. Thank you!
But it still doesn't work with client's by vault...

5 replies

scholzj Jun 5, 2022
Maintainer

It works with server crt and key and client's by strimzi. Thank you!
But it still doesn't work with client's by vault...

Ok, so if it doesn't work, then:

What does openssl s_client return when you try to connect to it
What does Java client return when you give it the option -Djavax.net.debug=ssl

Please share the full outputs and configurations.

akametall Jun 7, 2022
Author

vault write -field certificate kafka-int-ca/issue/kafka-client
common_name=my-client.clients.kafka.mts.ru format=pem_bundle > kafkaclient.pem
openssl pkcs12 -inkey kafkaclient.pem -in kafkaclient.pem -name kafkaclient -export -out kafkaclient.p12
keytool -importkeystore -deststorepass changeme -destkeystore kafkaclient.jks -srckeystore kafkaclient.p12 -srcstoretype PKCS12
security.protocol=SSL
ssl.keystore.location=/home/anivan54/kafka.client.keystore.jks
ssl.keystore.password=changeme
ssl.key.password=changeme
ssl.truststore.location=/home/anivan54/kafkaclietnt2.jks
ssl.truststore.password=changeme
ssl.endpoint.identification.algorithm:

5../bin/kafka-console-producer.sh --bootstrap-server ***:9094 -topic schemas --producer.config ./bin/client.properties

WARN [Producer clientId=console-producer] Bootstrap broker 10.73.96.58:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

I think i knew thats the problem. With only server certs i manually create user with user operator, and create acls for it. Like this:

piVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
name: kafka-admin
labels:
strimzi.io/cluster: prp-cluster
spec:
authentication:
type: tls
authorization:
type: simple
acls:
- resource:
type: topic
name: ""
patternType: literal
operation: All
host: ""

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: schemas
labels:
strimzi.io/cluster: prp-cluster
spec:
partitions: 1
replicas: 1
config:
retention.ms: -1

But hear... i actually don't know how to create user kafkaclient with my own client certs.

scholzj Jun 7, 2022
Maintainer

Well, tis is certainly not the right way. You will need to:

Follow this guide to install your own Clients CA: https://strimzi.io/docs/operators/latest/full/configuring.html#installing-your-own-ca-certificates-str (as long as you care only about clients, you can ignore the Cluster CA and doe only the Clients CA parts)
If you want to issue your certificates with Vault, you will need to provide both the ...clients-ca-cert and ...clients-ca secret. But you can just put some dummy file into the ...clients-ca secret.
Once you have thecustom Clients CA set-up, you can create the users using KafkaUSer resources using the type: tls-external authentication (https://strimzi.io/docs/operators/latest/full/configuring.html#tls_client_authentication_using_a_certificate_issued_outside_the_user_operator). That will make sure the User Operator manages the ACLs or the Quotas but does not issue the certificate.
Generate your clients certificate with Vault so that it is signed by the CA you used as Clients CA.

Keep in mind that the ssl.truststore.* options in the Kafka client relate to the server certificate of the Kafka broker. So they need to match what the broker uses as the Cluster CA or as the listener certificate. Only the ssl.keystore.* options should match the Clients CA / client certificate.

akametall Jun 7, 2022
Author

Thank you.
One question:
than i do kubectl describe secret i see in clients-ca some key file (you say i can put some dummy) and in clients-ca-cert .crt, p12, and password. But i only have pem...ok, i can transfer it to p12. But password and crt... Ok crt i can get from pem with ctrl-c ctrl v. Password just generate smth?

scholzj Jun 7, 2022
Maintainer

You can have a pem only. And if you want to generate the certs your self, you can just create it as a dummy file. It does not need to be a real private key.

akametall · 2022-06-08T10:57:15Z

akametall
Jun 8, 2022
Author

So now i create

kubectl create secret generic prp-cluster-clients-ca-cert --from-file=kafkaclient2.pem
kubectl create secret generic prp-cluster-clients-ca --from-file=kafkaclient2.key
kubectl create secret generic my-secret --from-file=prp-kafka-server.key --from-file=prp-kafka-server.crt

 brokerCertChainAndKey:
     secretName: my-secret
     certificate: prp-kafka-server.crt
     key: prp-kafka-server.key

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
name: kafka-admin
labels:
strimzi.io/cluster: prp-cluster
spec:
authentication:
type: tls-external
authorization:
type: simple
acls:
- resource:
type: topic
name: ""
patternType: literal
operation: All
host: ""
6.
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: schemas
labels:
strimzi.io/cluster: prp-cluster
spec:
partitions: 1
replicas: 1
config:
retention.ms: -1

and my question is what i must use as keystore and truststore

truststore is somth generateted with it --from-file=prp-kafka-server.key --from-file=prp-kafka-server.crt
but keystore...

1 reply

scholzj Jun 8, 2022
Maintainer

Please share the whole CRs so that I understand where way this placed in the Kafka CR.

akametall · 2022-06-08T13:30:03Z

akametall
Jun 8, 2022
Author

Kafka

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prp-cluster
spec:
  kafka:
    version: 3.0.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls
        configuration:
          brokerCertChainAndKey:
            secretName: my-secret
            certificate: prp-kafka-server.crt
            key: prp-kafka-server.key
          bootstrap:
            loadBalancerIP: smth
          brokers:
          - broker: 0
            loadBalancerIP: smth
    authorization:
      type: simple
#      superUsers:
#        - CN=kafka-admin
#        - CN=prp-admin
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - prp-cluster-kafka
                topologyKey: "kubernetes.io/hostname" 
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    config:
      num.partitions: 1
      default.replication.factor: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "3.0"
      inter.broker.protocol.version: "3.0"
      auto.create.topics.enable: false
#      log.retention.bytes: 1073741824
      log.retention.hours: 168
      log.retention.check.interval.ms: 300000
    resources:
      requests:
        memory: 1Gi
        cpu: "1"
      limits:
        memory: 1Gi
        cpu: "1"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 5Gi
        deleteClaim: false
#    metricsConfig:
#      type: jmxPrometheusExporter
#      valueFrom:
#        configMapKeyRef:
#          name: kafka-metrics
#          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 1
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - prp-cluster-zookeeper
                topologyKey: "kubernetes.io/hostname"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 500Mi
        cpu: 500m
    storage:
      type: persistent-claim
      size: 3Gi
      deleteClaim: false
  entityOperator:
    tlsSidecar:
      resources:
        requests:
          memory: 64Mi
          cpu: 100m
        limits:
          memory: 128Mi
          cpu: 150m
    topicOperator:
      resources:
        requests:
          memory: 100Mi
          cpu: 100m
        limits:
          memory: 300Mi
          cpu: 300m
    userOperator:
      resources:
        requests:
          memory: 100Mi
          cpu: 100m
        limits:
          memory: 300Mi
          cpu: 300m
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    resources:
      requests:
        cpu: 100m
        memory: 64Mi
      limits:
        cpu: 300m
        memory: 128Mi
    logging: debug
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5

8 replies

scholzj Jun 8, 2022
Maintainer

https://strimzi.io/docs/operators/latest/full/configuring.html#installing-your-own-ca-certificates-str

akametall Jun 8, 2022
Author

maybe i'm to stupid, but i don't see anything about

Clients CA configuration.

I generate it with vault and create secrets with it. Like in docs.
and generate the keystore with it. That else?

scholzj Jun 8, 2022
Maintainer

The whole CA configuration section is what you are missing in the Kafka CR.

akametall Jun 8, 2022
Author

like this?

kind: Pod
apiVersion: v1
metadata:
  name: client-pod
spec:
  containers:
  - name: client-name
    image: client-name
    volumeMounts:
    - name: secret-volume
      mountPath: /data/p12
    env:
    - name: SECRET_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: my-password
  volumes:
  - name: secret-volume
    secret:
      secretName: my-cluster-cluster-ca-cert

Sorry, but i didn't see anything about configuring kafka deployment except this

  clusterCa:
    generateCertificateAuthority: false

scholzj Jun 8, 2022
Maintainer

There is a clientsCa as well -> that is what you need to use. But I'm not completely sure how to change it for exsiting cluster.

Strimzi

Need help about installing my own cert and key #6867

Uh oh!

akametall May 25, 2022

Replies: 14 comments · 26 replies

Uh oh!

scholzj May 25, 2022 Maintainer

Uh oh!

akametall May 25, 2022 Author

Uh oh!

akametall May 25, 2022 Author

Uh oh!

scholzj May 25, 2022 Maintainer

Uh oh!

akametall May 25, 2022 Author

Uh oh!

scholzj May 25, 2022 Maintainer

Uh oh!

Uh oh!

akametall May 26, 2022 Author

Uh oh!

akametall May 26, 2022 Author

Uh oh!

scholzj May 26, 2022 Maintainer

Uh oh!

Uh oh!

akametall May 26, 2022 Author

Uh oh!

Uh oh!

scholzj May 26, 2022 Maintainer

Uh oh!

akametall May 27, 2022 Author

Uh oh!

scholzj May 27, 2022 Maintainer

Uh oh!

akametall May 27, 2022 Author

Uh oh!

scholzj May 27, 2022 Maintainer

Uh oh!

akametall May 29, 2022 Author

Uh oh!

akametall May 29, 2022 Author

- broker: 1

loadBalancerIP:

- broker: 2

loadBalancerIP: fff

superUsers:

- CN=kafka-admin

Uh oh!

akametall May 29, 2022 Author

Uh oh!

akametall May 29, 2022 Author

Uh oh!

akametall May 29, 2022 Author

Uh oh!

akametall May 29, 2022 Author

Uh oh!

akametall May 29, 2022 Author

Uh oh!

akametall May 29, 2022 Author

Uh oh!

akametall May 31, 2022 Author

Uh oh!

scholzj Jun 5, 2022 Maintainer

Uh oh!

akametall Jun 7, 2022 Author

Uh oh!

scholzj Jun 7, 2022 Maintainer

Uh oh!

akametall Jun 7, 2022 Author

Uh oh!

scholzj Jun 7, 2022 Maintainer

Uh oh!

akametall Jun 8, 2022 Author

akametall
May 25, 2022

Replies: 14 comments 26 replies

scholzj
May 25, 2022
Maintainer

akametall May 25, 2022
Author

akametall May 25, 2022
Author

scholzj May 25, 2022
Maintainer

akametall May 25, 2022
Author

scholzj May 25, 2022
Maintainer

akametall
May 26, 2022
Author

akametall May 26, 2022
Author

scholzj May 26, 2022
Maintainer

akametall
May 26, 2022
Author

scholzj May 26, 2022
Maintainer

akametall May 27, 2022
Author

scholzj May 27, 2022
Maintainer

akametall May 27, 2022
Author

scholzj May 27, 2022
Maintainer

akametall
May 29, 2022
Author

akametall
May 29, 2022
Author

akametall
May 29, 2022
Author

akametall
May 29, 2022
Author

akametall
May 29, 2022
Author

akametall
May 29, 2022
Author

akametall
May 29, 2022
Author

akametall
May 29, 2022
Author

akametall
May 31, 2022
Author

scholzj Jun 5, 2022
Maintainer

akametall Jun 7, 2022
Author

scholzj Jun 7, 2022
Maintainer

akametall Jun 7, 2022
Author

scholzj Jun 7, 2022
Maintainer

akametall
Jun 8, 2022
Author