input: add thread.ring_buffer.retry_limit option #11394

jinyongchoi · 2026-01-24T04:16:30Z

When a threaded input plugin's ring buffer is full, the input thread retries writing to the buffer before dropping data. Previously, this retry limit was hardcoded to 10 (1 second total with 100ms sleep between retries).

This patch makes the retry limit configurable via the new 'thread.ring_buffer.retry_limit' option. The default value remains 10 for backward compatibility. Increasing this value allows the system to handle temporary backpressure situations without dropping data.

Fixes #11393

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

[SERVICE]
    flush 1
    grace 600
    log_level info

    storage.path /tmp/storage
    storage.metrics on
    storage.max_chunks_up 512
    storage.sync full
    storage.checksum off
    storage.backlog.mem_limit 100M

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag app_logs
    Key message

    Read_from_Head true
    Refresh_Interval 3

    Buffer_Chunk_Size 1MB
    Buffer_Max_Size 16MB

    Mem_Buf_Limit 256MB
    storage.type memory
    storage.pause_on_chunks_overlimit true

    DB /tmp/storage/tail.db
    DB.sync normal

    Threaded true
    thread.ring_buffer.capacity  128
    thread.ring_buffer.retry_limit  1000

[OUTPUT]
    Name kafka
    Match app_logs
    Brokers kafka:9092
    Topics logs
    timestamp_format iso8601

    rdkafka.compression.codec gzip
    rdkafka.compression.level 7
    rdkafka.queue.buffering.max.messages 200000
    rdkafka.batch.size 1000000
    rdkafka.linger.ms 100

    workers 4

[N/A] Debug log output from testing the change
[N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

New Features
- Configurable retry limit for threaded input ring-buffer operations via new setting thread.ring_buffer.retry_limit (default: 10). Users can control how many times writes are retried when the ring buffer is full.
Bug Fixes / Observability
- Enqueue failures now increment a failure metric and the configured retry value is validated (>0).

_{✏️ Tip: You can customize this high-level summary in your review settings.}

coderabbitai · 2026-01-24T04:17:00Z

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

Adds a per-input configurable retry limit for writing to the threaded input ring buffer: new field ring_buffer_retry_limit on the input instance, config parsing and default in input initialization, and usage of the instance field in enqueue retry logic.

Changes

Cohort / File(s)	Summary
Input struct definition `include/fluent-bit/flb_input.h`	Added `int ring_buffer_retry_limit` field to `struct flb_input_instance`.
Configuration and initialization `src/flb_input.c`	Added `FLB_INPUT_RING_BUFFER_RETRY_LIMIT` macro (default 10); registered `thread.ring_buffer.retry_limit` in input properties; initialize `ins->ring_buffer_retry_limit` to default; parse, validate (>0) and assign property in `flb_input_set_property`.
Ring buffer retry implementation `src/flb_input_chunk.c`	Replaced local hardcoded retry limit with `ins->ring_buffer_retry_limit` in ring buffer enqueue retry logic; increment failure counter on enqueue failure.

Sequence Diagram(s)

(Skipped — changes are focused and do not introduce a multi-component sequential flow requiring visualization.)

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐇 I nudged a limit, soft and slight,
Now threads may try long into the night.
Rings hum gently, retries made true,
Little drops kept — a happier crew. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: adding a configurable thread.ring_buffer.retry_limit option to threaded input plugins.
Linked Issues check	✅ Passed	All coding requirements from issue `#11393` are met: the option thread.ring_buffer.retry_limit is implemented with default value 10, applied per-input instance, and validated for values > 0.
Out of Scope Changes check	✅ Passed	All changes directly implement the feature requested in `#11393` with no unrelated modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

This PR adds a configurable retry limit for threaded input plugins' ring buffer writes. Previously, the retry limit was hardcoded to 10 attempts (1 second total with 100ms sleep between retries). This caused data loss during temporary backpressure situations that lasted longer than 1 second.

Changes:

Added thread.ring_buffer.retry_limit configuration option with default value of 10 to maintain backward compatibility
Modified ring buffer write logic to use the configurable retry limit instead of hardcoded value
Updated documentation to explain the new configuration option

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
include/fluent-bit/flb_input.h	Added `ring_buffer_retry_limit` field to `flb_input_instance` struct
src/flb_input.c	Added configuration map entry, initialization, and property parsing for the new retry limit option
src/flb_input_chunk.c	Removed hardcoded retry limit and updated to use instance-level configurable value

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-24T04:19:34Z

src/flb_input.c

+    else if (prop_key_check("thread.ring_buffer.retry_limit", k, len) == 0 && tmp) {
+        ret = atoi(tmp);
+        flb_sds_destroy(tmp);
+        if (ret <= 0) {
+            flb_error("[input] thread.ring_buffer.retry_limit must be greater than 0");
+            return -1;
+        }
+        ins->ring_buffer_retry_limit = ret;
+    }


Consider adding an upper bound validation for the retry_limit to prevent potential issues with excessively large values.

With the 100ms sleep between retries, a very large retry_limit could cause the input thread to block for an extended period. For example, a value of 1,000,000 would result in approximately 27.7 hours of blocking. This could cause unexpected behavior or resource exhaustion.

A reasonable upper bound might be something like 36000 (1 hour of retries) or 600 (1 minute), with a warning message if the value exceeds a certain threshold. The appropriate limit should depend on your system's requirements and typical backpressure patterns.

I intentionally chose not to enforce an upper bound because the primary use case for this option is environments where data loss is unacceptable (e.g., audit logs, financial transactions).

In such cases, users prefer the input thread to block and wait for backpressure to resolve rather than dropping data. The user explicitly configures a large value knowing the trade-off.

Add a new configuration option 'thread.ring_buffer.retry_limit' for threaded input plugins. This option controls the maximum number of retry attempts when the ring buffer is full before dropping data. The default value is 10 (maintaining backward compatibility). Fixes fluent#11393 Signed-off-by: jinyong.choi <inimax801@gmail.com>

Replace the hardcoded retry limit (10) with the configurable 'ring_buffer_retry_limit' value from the input instance. This allows users to increase retry attempts for handling temporary backpressure situations without dropping data. Signed-off-by: jinyong.choi <inimax801@gmail.com>

jinyongchoi · 2026-01-24T06:35:22Z

@codex review

chatgpt-codex-connector · 2026-01-24T06:39:27Z

Codex Review: Didn't find any major issues. 👍

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Copilot AI review requested due to automatic review settings January 24, 2026 04:16

jinyongchoi requested review from cosmo0920 and edsiper as code owners January 24, 2026 04:16

github-actions bot added the docs-required label Jan 24, 2026

jinyongchoi temporarily deployed to pr January 24, 2026 04:16 — with GitHub Actions Inactive

Copilot started reviewing on behalf of jinyongchoi January 24, 2026 04:16 View session

Copilot AI reviewed Jan 24, 2026

View reviewed changes

jinyongchoi added 2 commits January 24, 2026 13:29

jinyongchoi force-pushed the feature/input-ring-buffer-retry-limit branch from 0e15cc1 to dba41e4 Compare January 24, 2026 04:29

jinyongchoi temporarily deployed to pr January 24, 2026 04:30 — with GitHub Actions Inactive

jinyongchoi temporarily deployed to pr January 24, 2026 04:49 — with GitHub Actions Inactive

jinyongchoi had a problem deploying to pr January 24, 2026 04:49 — with GitHub Actions Failure

jinyongchoi temporarily deployed to pr January 24, 2026 04:49 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

input: add thread.ring_buffer.retry_limit option #11394

input: add thread.ring_buffer.retry_limit option #11394

jinyongchoi commented Jan 24, 2026 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Jan 24, 2026 •

edited

Loading

Other AI code review bot(s) detected

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Poem

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 24, 2026

Uh oh!

jinyongchoi Jan 24, 2026

Uh oh!

jinyongchoi commented Jan 24, 2026

Uh oh!

chatgpt-codex-connector bot commented Jan 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

input: add thread.ring_buffer.retry_limit option #11394

Are you sure you want to change the base?

input: add thread.ring_buffer.retry_limit option #11394

Conversation

jinyongchoi commented Jan 24, 2026 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Other AI code review bot(s) detected

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Poem

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 24, 2026

Choose a reason for hiding this comment

Uh oh!

jinyongchoi Jan 24, 2026

Choose a reason for hiding this comment

Uh oh!

jinyongchoi commented Jan 24, 2026

Uh oh!

chatgpt-codex-connector bot commented Jan 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jinyongchoi commented Jan 24, 2026 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Jan 24, 2026 •

edited

Loading