[Q&A] fluentd-kafka-plugin. Enable SASL with OAUTHBEARER mechanism provided by KeyCloak

[yogrr]

What is a problem?

I am unable to connect to a Kafka broker using the fluentd-plugin-kafka with SASL_SSL and the OAUTHBEARER mechanism. The Fluentd container fails to start the Kafka source plugin due to a missing dependency in the underlying rdkafka-ruby library.

This configuration is correct according to the rdkafka-ruby documentation. However, when the Fluentd container starts, it logs an error indicating that the build of the rdkafka-ruby gem does not support OAuth. This suggests that the Docker image I'm using for Fluentd was built without the necessary libcurl and OpenSSL development headers, which are required to compile the librdkafka C library with OAuth support.

The issue is not with the Fluentd plugin's configuration, but with the environment and dependencies in which the rdkafka-ruby gem was installed.

Describe the configuration of Fluentd

<source>
      @type rdkafka_group
      @id infra_logs_kafka_source
      @label @INFRA_LOGS

      kafka_configs {
        "bootstrap.servers": "brokers #{ENV['KAFKA_BROKERS']}",
        "group.id": "fluentd-infra-logs",

        "security.protocol": "sasl_ssl",

        "sasl.mechanisms": "OAUTHBEARER",
        "sasl.oauthbearer.client.id": "kafka-logs",
        "sasl.oauthbearer.client.secret": "XXXsecretXXX",
        "sasl.oauthbearer.token.endpoint.url": "https://keycloak.url/realms/space/protocol/openid-connect/token"
      }

      topics infra-logs-topic

      format json

      add_prefix kafka.infra
      add_suffix logs
</source>

Describe the logs of Fluentd

...
{"time":"2025-09-09 19:44:18 +0000","level":"debug","message":"[infra_logs_opensearch] enqueue_thread actually running","worker_id":0}
{"time":"2025-09-09 19:44:18 +0000","level":"error","message":"unexpected error error_class=Rdkafka::Config::ConfigError error=\"Configuration property \\\"sasl.oauthbearer.client.id\\\" not supported in this build: OAuth/OIDC depends on libcurl and OpenSSL which were not available at build time\"","worker_id":0}

{
  "time": "2025-09-09 19:44:18 +0000",
  "level": "error",
  "worker_id": 0,
  "message": "/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.23.0-x86_64-linux-gnu/lib/rdkafka/config.rb:317:in `block (2 levels) in native_config'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.23.0-x86_64-linux-gnu/lib/rdkafka/config.rb:307:in `each'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.23.0-x86_64-linux-gnu/lib/rdkafka/config.rb:307:in `block in native_config'\n<internal:kernel>:90:in `tap'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.23.0-x86_64-linux-gnu/lib/rdkafka/config.rb:305:in `native_config'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.23.0-x86_64-linux-gnu/lib/rdkafka/config.rb:205:in `consumer'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.5/lib/fluent/plugin/in_rdkafka_group.rb:164:in `setup_consumer'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.5/lib/fluent/plugin/in_rdkafka_group.rb:147:in `start'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/root_agent.rb:203:in `block in start'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/root_agent.rb:191:in `each'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/root_agent.rb:191:in `block in lifecycle'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1 /lib/fluent/root_agent.rb:178:in `each'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/root_agent.rb:178:in `lifecycle'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/root_agent.rb:202:in `start'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/engine.rb:248:in `start'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/engine.rb:147:in `run'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:617:in `block in run_worker'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:962:in `main _process'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:608:in `run_worker'\n/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/lib/fluent/command/fluentd.rb:372:in `<top (required)>'\n<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'\n<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'\n/fluentd /vendor/bundle/ruby/3.2.0/gems/fluentd-1.17.1/bin/fluentd:15:in `<top (required)>'\n/fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `load'\n/fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `<main>'"
}
...

Environment

- Fluentd version: fluentd 1.17.1
- TD Agent version: 
- Fluent Package version: [v0.19.5](https://github.com/fluent/fluent-plugin-kafka/releases/tag/v0.19.5)
- Docker image (tag): v1.17.1-debian-opensearch-1.2 (fluent/fluentd-kubernetes-daemonset:v1.17.1-debian-opensearch-1.2)
- Operating system: debian bookworm
- Kernel version: 6.6.28-1-generic

- Installed Gems versions:
   - fluent-plugin-kafka (0.19.5)
   - rdkafka (0.23.0 x86_64-linux-gnu)
   - ruby-kafka (1.5.0)

[yogrr]

I understand that the librdkafka library is built without OpenSSL support, but how do I install the rdkafka-ruby gem with librdkafka, which includes this support?