[Q&A] Certificate error on connection to OCI broker

[JGouma]

What is a problem?

I try to ingest logging from OCI streaming. When starting get the following error in fluentd.log:

2025-09-09 09:40:44 +0200 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 peeraddr=**10.55.xx.xx**:9092 state=error: certificate verify failed (unable to get issuer certificate)"
2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/ssl_socket_with_timeout.rb:69:in 'OpenSSL::SSL::SSLSocket#connect_nonblock'
2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/ssl_socket_with_timeout.rb:69:in 'Kafka::SSLSocketWithTimeout#initialize'
2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/connection.rb:130:in 'Class#new'
2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/connection.rb:130:in 'Kafka::Connection#open'

The ip number show (10.55.xx.xx) is the internal IP-numebr of the broker which is mentionend by FQDN in the fluentd.conf file.

For some reason fluentd seems to verify the certificate with the ip-number instead of the hostname.

Accoding openssl the certificate is valid:

[root@tst-fluentd-01 fluent]# openssl verify kafka_chain2.pem
kafka_chain2.pem: OK
[root@tst-fluentd-01 fluent]#

Also an openssl connection to the broker gives a valid certificate:

[root@tst-fluentd-01 fluent]# openssl s_client -showcerts -connect xxxxxxx.streaming.eu-amsterdam-1.oci.oraclecloud.com:9092
Connecting to 10.55.xx.xxx
CONNECTED(00000003)
depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
verify return:1
depth=1 C=US, O=DigiCert Inc, CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C=US, ST=California, L=Redwood City, O=Oracle Corporation, CN=streaming.eu-amsterdam-1.oci.oraclecloud.com
verify return:1
---
Certificate chain
 0 s:C=US, ST=California, L=Redwood City, O=Oracle Corporation, CN=streaming.eu-amsterdam-1.oci.oraclecloud.com
   i:C=US, O=DigiCert Inc, CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov  1 00:00:00 2024 GMT; NotAfter: Nov  5 23:59:59 2025 GMT

Why is fluentd verifying against the IP instead of the actual hostname? How to solve this?

Thanks a lot for any help!

Jeroen

Describe the configuration of Fluentd

<source>
  @type kafka_group
  brokers "xxxxxxxxxx.streaming.eu-amsterdam-1.oci.oraclecloud.com:9092"
  topics TST_Audit_log
  format json
  username tenant/username/ocid1.streampool.oc1.eu-amsterdam-1.amaaaaaac5s3XXXXXXXXXXXXXXXXm3l5a
  password <redacted>
  ssl_ca_cert /etc/fluent/kafka_chain.pem
  sasl_over_ssl true
  consumer_group fluentd-group
  <parse>
        @type json
  </parse>
</source>

Describe the logs of Fluentd

2025-09-09 09:40:44 +0200 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 peeraddr=10.55.xx.xx:9092 state=error: certificate verify failed (unable to get issuer certificate)"
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/ssl_socket_with_timeout.rb:69:in 'OpenSSL::SSL::SSLSocket#connect_nonblock'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/ssl_socket_with_timeout.rb:69:in 'Kafka::SSLSocketWithTimeout#initialize'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/connection.rb:130:in 'Class#new'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/connection.rb:130:in 'Kafka::Connection#open'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/connection.rb:101:in 'block in Kafka::Connection#send_request'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/instrumenter.rb:23:in 'Kafka::Instrumenter#instrument'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/connection.rb:100:in 'Kafka::Connection#send_request'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/sasl_authenticator.rb:65:in 'Kafka::SaslAuthenticator#authenticate!'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/connection_builder.rb:27:in 'Kafka::ConnectionBuilder#build_connection'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/broker.rb:214:in 'Kafka::Broker#connection'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/broker.rb:200:in 'Kafka::Broker#send_request'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/broker.rb:44:in 'Kafka::Broker#fetch_metadata'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/cluster.rb:432:in 'block (2 levels) in Kafka::Cluster#fetch_cluster_info'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/cluster.rb:425:in 'Array#each'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/cluster.rb:425:in 'block in Kafka::Cluster#fetch_cluster_info'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/cluster.rb:424:in 'Array#each'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/cluster.rb:424:in 'Kafka::Cluster#fetch_cluster_info'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/cluster.rb:405:in 'Kafka::Cluster#cluster_info'
  2025-09-09 09:40:44 +0200 [error]: #0 /opt/fluent/lib/ruby/gems/3.4.0/gems/ruby-kafka-1.5.0/lib/kafka/cluster.rb:105:in 'Kafka::Cluster#refresh_metadata!'

Environment

- fluent-package 6.0.0 fluentd 1.19.0 (ed15edc4b7c86c1fcea3fbf326b7c42325e6fc71)
- Operating system: Oracle Linux 9.6
- Installed gems: 
  - fluent-plugin-kafka 
  - fluent-plugin-elasticsearch

[JGouma]

After more searching found the following solution:

Add the setting "ssl_ca_certs_from_system true" to fluend.conf.

Solved this issue