[Q&A] Resend backed up buffer files that are too big for elastic to handle

[addyy505]

What is a problem?

Hello people! We had the problem that due to misconfiguration of fluentd's elasticsearch output plugin from our developers, we had to backup all buffer files from /var/log/fluentd/workeri/elasticsearch/buffer/*.log|*.meta to /var/log/fluentd/backup/workeri/elasticsearch/buffer/*.log|*.meta because it was a in prod stage an logs were not being send due to the chunk_limit_size config being too big and elastic was not able to accepte the large request. We changed the configuartion from

<buffer>
  @type file
  chunk_limit_size 128MB
  flush_mode interval
  flush_interval 10s
  flush_thread_count 8
  flush_at_shutdown true
  retry_type exponential_backoff
  retry_max_interval 3600s
  retry_forever true
  overflow_action throw_exception
</buffer>

to

bulk_message_request_threshold 64M

<buffer>
  @type file
  chunk_limit_size 8MB
  flush_mode interval
  flush_interval 10s
  flush_thread_count 8
  flush_at_shutdown true
  retry_type exponential_backoff
  retry_max_interval 3600s
  retry_forever true
  overflow_action block
</buffer>

and this works. But the problem now is we are contractually required to send all logs. How do I resend the backed up chunks even though they are too big? Thank you for reading and your help in advance!

Describe the configuration of Fluentd

We changed

<buffer>
  @type file
  chunk_limit_size 128MB
  flush_mode interval
  flush_interval 10s
  flush_thread_count 8
  flush_at_shutdown true
  retry_type exponential_backoff
  retry_max_interval 3600s
  retry_forever true
  overflow_action throw_exception
</buffer>

to

bulk_message_request_threshold 64M

<buffer>
  @type file
  chunk_limit_size 8MB
  flush_mode interval
  flush_interval 10s
  flush_thread_count 8
  flush_at_shutdown true
  retry_type exponential_backoff
  retry_max_interval 3600s
  retry_forever true
  overflow_action block
</buffer>

Describe the logs of Fluentd

No response

Environment

- Fluentd version: 1.15
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system: Alpine Linux v3.13
- Kernel version: 6.1.0-31-amd64