Failed to construct kafka producer

[stuartm21]

Hi @scholzj ,
We have a kafka cluster with 3.1.0 version and enabled MTLS for that. Now we are trying to connect logstash with kafka and we are getting the following errors from logstash logs. The keystore and truststore are accessible through the password.

tail /var/log/logstash/logstash-plain.log 
    value.serializer = class org.apache.kafka.common.serialization.StringSerializer

[2022-04-04T05:29:21,635][INFO ][org.apache.kafka.clients.producer.KafkaProducer][main] [Producer clientId=test-box] Closing the Kafka producer with timeoutMillis = 0 ms.
[2022-04-04T05:29:21,649][ERROR][logstash.outputs.kafka   ][main] Unable to create Kafka producer from given configuration {:kafka_error_message=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer, :cause=>org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/logstash/certs/truststore.jks of type JKS}
[2022-04-04T05:29:21,663][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer, :backtrace=>["org.apache.kafka.clients.producer.KafkaProducer.<init>(org/apache/kafka/clients/producer/KafkaProducer.java:432)", "org.apache.kafka.clients.producer.KafkaProducer.<init>(org/apache/kafka/clients/producer/KafkaProducer.java:298)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)",

logstash conf:

cat 90-output.conf 
output {
  kafka {
    bootstrap_servers => "kafka-bootstrap.myhost:443"
    topic_id => "kafka-topic"
    security_protocol => "SSL"
    client_id => "test-box"
    ssl_endpoint_identification_algorithm => "https"
    ssl_truststore_location => "/etc/logstash/certs/truststore.jks"
    ssl_truststore_password => "h02uoyuuB284"
    ssl_keystore_location => "/etc/logstash/certs/keystore.jks"
    ssl_keystore_password => "h02uoyuuB284"
    ssl_key_password => "h02uoyuuB284"
    codec => "json"
  }
}

The truststore.jks and keystore.jks are having 644 linux permission and the ownership(user&group) is logstash. And the truststore.jks and keystore.jks created using the following commands:

kubectl -n kafka get secret strimzi-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl -n kafka get secret kafka-user -o jsonpath='{.data.user\.password}' | base64 -d > user.password
kubectl -n kafka get secret kafka-user -o jsonpath='{.data.user\.p12}' | base64 -d > user.p12

export PASSWORD=`cat user.password`
keytool -keystore truststore.jks -alias CARoot -import -file ca.crt -storepass $PASSWORD -noprompt
keytool -importkeystore -srckeystore user.p12 -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype jks 

[scholzj]

Sorry, I cannot help with this. Your steps to create the keystore / truststore look good to me.

[stuartm21]

@scholzj Ok I understood. At least can you tell me is this kafka issue or logstash issue.?

[scholzj]

I don't know Logstash -> but the error suggests it cannot properly read your keystore which would have nothing to do with the Kafka cluster.