Fluentd continues to commit kafka offset even if output is down

[hassanbenhzaine]

Hello everyone,

I have "fluent-plugin-kafka" (Input plugin) as source sending Kafka records to "fluent-plugin-elasticsearch" (Output plugin) and it works fine, the issue is if Elasticsearch is down Fluentd continues to commit offsets (consuming messages) and retry connecting to Elasticsearch, is there anyway to periodically check if Elasticsearch is online before consuming records from Kafka?

// fluent.conf file

<source>
  @type kafka_group

  brokers localhost:9092
  consumer_group group4
  topics new_orders1
  format json
  start_from_beginning  true
  add_headers true
</source>

<match new_orders1>
  @type elasticsearch
  host localhost
  port 9200
  index_name fluentd
  type_name fluentd
</match>

[cosmo0920]

We don't recommend to use healthcheck to confirm the service availability.
Instead, we recommend to use file buffer plugin to prevent events lost:
https://docs.fluentd.org/configuration/buffer-section#buffer-plugin-type

[hassanbenhzaine]

I've been looking into creating my own output plugin, this one will get fluentd logs and parse them, using those logs i can then decide to stop or resume kafka plugin thread

The downside is i had to modify the kafka input plugin to give the thread a name

// out_kafka_group.rb

// fluent.conf

What do you think?