config: storage: Add total limit size of DLQ

[cosmo0920]

Currently, there's no way to specify the total size of limit on DLQ.
This PR adds this capability to limit infinitely growing the total amount of DLQ.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Added a configuration option to set a maximum size for the storage dead-letter queue (DLQ), allowing users to cap bytes used for rejected messages.

• Bug Fixes

  • DLQ now enforces the configured size limit: it prevents creating DLQ entries once the limit is reached or would be exceeded and logs a warning when usage approaches the limit.

• Tests

  • Added a test verifying the DLQ respects the configured size limit.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 86b8908e-7c94-4616-8105-8f28232f17a1

📥 Commits

Reviewing files that changed from the base of the PR and between 41528be and d63c537.

📒 Files selected for processing (1)

• src/flb_storage.c

🚧 Files skipped from review as they are similar to previous changes (1)

• src/flb_storage.c

---

📝 Walkthrough

Walkthrough

Adds a storage.rejected.limit config, wires it into flb_config, enforces a total DLQ (rejected stream) size limit in flb_storage_quarantine_chunk(), and adds a test ensuring no DLQ file is written when the limit would be exceeded.

Changes

Storage DLQ Size Limit

Layer / File(s)	Summary
Configuration schema and wiring include/fluent-bit/flb_config.h, src/flb_config.c	Adds storage_rejected_limit to struct flb_config, defines FLB_CONF_STORAGE_REJECTED_LIMIT, wires it in service_configs[], initializes it to NULL, and frees it on exit.
DLQ size limit enforcement in quarantine src/flb_storage.c	Adds <inttypes.h>, implements dlq_stream_total_size() to compute current DLQ bytes, parses storage_rejected_limit, and makes flb_storage_quarantine_chunk() skip DLQ writes when the existing total or the post-add total would exceed the configured limit (with warnings near 90%).
Test coverage for DLQ size limit tests/internal/storage_dlq.c	Adds test_dlq_respects_size_limit() that sets a small DLQ limit, attempts to quarantine an oversized chunk, asserts failure and absence of a rejected/DLQ file, and registers the test.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• fluent/fluent-bit#11000: Introduced the DLQ quarantine mechanism which this PR extends by adding a configurable total DLQ size limit and enforcement.

Suggested reviewers

• edsiper
• patrick-stephens
• fujimotos

Poem

🐰 A little rabbit hops through code tonight,
Counting bytes in a DLQ ever so tight,
"Storage limit set!" the config does sing,
Quarantine skips when the totals take wing,
Tests nod and the caught rejections rest light.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'config: storage: Add total limit size of DLQ' directly and accurately describes the main change: introducing a configuration property to set a total size limit for the Dead Letter Queue storage.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cosmo0920-add-total-limit-size-of-dlq

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 41528bebc6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/flb_storage.c`:
- Around line 847-864: The size accounting bug: dlq_stream_total_size()
currently sums cio_chunk_get_real_size() (including file overhead) but the limit
check at the write path uses the chunk content size from
cio_chunk_get_content_copy(), allowing the actual on-disk size to exceed the
configured limit; fix by changing dlq_stream_total_size() to sum chunk content
sizes instead of real sizes (use the same content-size accessor used at the
check—or call cio_chunk_get_content_copy()/its size accessor or an equivalent
function that returns payload bytes) so both the pre-write check and the
current-total computation use the same unit (content bytes); update any callers
or comments accordingly to reflect that the DLQ limit applies to content size.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0c307eb8-4aa8-4b96-a7f5-2981935e8beb

📥 Commits

Reviewing files that changed from the base of the PR and between 7299905 and 41528be.

📒 Files selected for processing (4)

• include/fluent-bit/flb_config.h
• src/flb_config.c
• src/flb_storage.c
• tests/internal/storage_dlq.c