consolidate backpressure/buffering info #2324
Conversation
Signed-off-by: Alexa Kreizinger <[email protected]>
📝 WalkthroughWalkthroughSplit and reorganized the former "Buffering and Storage" doc into focused pages: Changes
Sequence Diagram(s)(Skipped — documentation reorganization does not introduce multi-component control-flow changes requiring a sequence diagram.) Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Fix all issues with AI agents
In `@administration/backpressure.md`:
- Line 63: The link text in administration/backpressure.md is misspelled: change
the bracketed link label "[serivce section]" to "[service section]" so the
markdown link to
../administration/configuring-fluent-bit/yaml/service-section.md resolves
correctly; update only the label text in the link while keeping the target path
unchanged.
- Line 16: The link in the sentence uses an invalid double-hash anchor
"../pipeline/buffering.md#buffering-modes#filesystem-buffering"; replace it with
a single valid anchor (e.g., "../pipeline/buffering.md#filesystem-buffering") so
the Markdown link points to the correct section; update the anchor string in the
file to remove the extra "#" and ensure it matches the target heading slug in
pipeline/buffering.md.
- Line 59: Fix the Markdown link on the line that currently reads "[filesystem
buffering](../pipeline/buffering.md#buffering-modes#memory-only-buffering)" by
removing the duplicated '#' and pointing it to the filesystem buffering anchor
so link text matches target (for example replace with "[filesystem
buffering](../pipeline/buffering.md#buffering-modes-filesystem-buffering)" or
the correct "#filesystem-buffering" anchor used in that file).
- Line 13: Replace the broken double-hash anchor links in the documentation:
change occurrences of "#buffering-modes#memory-only-buffering" to
"#memory-only-buffering", change "#buffering-modes#filesystem-buffering" to
"#filesystem-buffering-hybrid", and update the second occurrence of
"#buffering-modes#memory-only-buffering" (in the filesystem buffering section)
to "#filesystem-buffering-hybrid" so all anchor targets point to the correct
headings.
In `@administration/configuring-fluent-bit/yaml/service-section.md`:
- Line 40: The Markdown link for Monitoring in the `storage.metrics` description
is using a relative path that won't resolve from the
administration/configuring-fluent-bit/yaml/ directory; update the
`[Monitoring](monitoring.md)` link to use the correct two-level-up relative path
(`../../monitoring.md`) so the link resolves to the `monitoring.md` file in the
administration directory, modifying the line that contains `storage.metrics` /
the `[Monitoring]` link.
In `@administration/dead-letter-queue.md`:
- Line 5: Update the two relative links that currently use a double slash prefix
(e.g.,
".//configuring-fluent-bit/yaml/service-section.md#storage-configuration") to
use a single slash ("/configuring-fluent-bit/...") so they resolve correctly;
specifically edit the link targets surrounding the `storage.path` and
`storage.keep.rejected` references in the dead-letter-queue text to remove the
extra slash.
In `@pipeline/buffering.md`:
- Around line 82-83: Change the wording in the `storage.path` description to use
the singular "service" (replace "services" with "service") to match Fluent Bit
terminology, and correct the unit-size link reference by adjusting the relative
path from `../../administration/configuring-fluent-bit.md#unit-sizes` to
`../administration/configuring-fluent-bit.md#unit-sizes` so the `mem_buf_limit`
unit-size link resolves correctly; update the table cell that mentions
`storage.path` and the `mem_buf_limit` link accordingly.
- Line 141: The markdown link for `storage.total_limit_size` uses the wrong
relative path (`../../administration/configuring-fluent-bit.md`); update that
link to `../administration/configuring-fluent-bit.md` in the table row
containing `storage.total_limit_size` so it correctly points to the
administration doc from this file's location.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
.gitbook.yamlSUMMARY.mdadministration/backpressure.mdadministration/buffering-and-storage.mdadministration/configuring-fluent-bit/yaml/service-section.mdadministration/dead-letter-queue.mdadministration/memory-management.mdpipeline/buffering.mdpipeline/inputs/tail.md
💤 Files with no reviewable changes (1)
- administration/buffering-and-storage.md
🧰 Additional context used
🪛 LanguageTool
administration/backpressure.md
[grammar] ~63-~63: Ensure spelling is correct
Context: ...g.mem_limit in global settings In the [serivce` section](../administration/configuring-...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
administration/dead-letter-queue.md
[style] ~43-~43: Consider using a different verb for a more formal wording.
Context: ...play or transform rejected chunks after fixing the underlying issue. - Debugging: ...
(FIX_RESOLVE)
🔇 Additional comments (11)
.gitbook.yaml (1)
106-106: LGTM!The redirect correctly maps the old
administration/buffering-and-storagepath to the new./pipeline/buffering.mdlocation, ensuring existing links and bookmarks continue to work after the documentation restructure.administration/dead-letter-queue.md (1)
1-89: Well-structured DLQ documentation.The document provides comprehensive coverage of the dead letter queue feature including failure scenarios, file location/format, troubleshooting guidance, and configuration examples in both YAML and CONF formats.
administration/configuring-fluent-bit/yaml/service-section.md (1)
34-47: LGTM on storage configuration updates.The expanded storage key descriptions are clear and comprehensive. The cross-references to the new Buffering, Backpressure, and Scheduling documentation align well with the PR's restructuring goals.
pipeline/buffering.md (1)
29-63: Well-documented buffering concepts.The new Chunks section clearly explains the
up/downchunk terminology and state transitions. The Buffering modes section effectively communicates the trade-offs between memory-only and filesystem buffering approaches.pipeline/inputs/tail.md (1)
79-79: LGTM!The cross-references are correctly updated to point to the new modular documentation structure, linking to both the Buffering guide and the Backpressure guide instead of the old combined document.
SUMMARY.md (1)
59-59: Navigation update looks good.The Table of Contents correctly reflects the new documentation structure, with Dead Letter Queue now as a standalone topic under Administration.
administration/memory-management.md (2)
11-11: Good technical clarification.The expanded explanation about internal binary representation and output plugin memory usage provides valuable context for memory estimation, with concrete examples.
15-17: Excellent cross-reference addition.The info hint provides helpful navigation to related Backpressure documentation, improving discoverability for readers learning about
Mem_Buf_Limit.administration/backpressure.md (3)
5-10: Clear introduction to backpressure concepts.The introduction effectively explains the backpressure condition and appropriately references the buffering documentation, setting up the mode-specific guidance that follows.
27-56: Detailed behavior explanation is helpful.The step-by-step example clearly illustrates what happens when
mem_buf_limitcapacity is reached, including the pause/resume mechanism. This will help users understand backpressure management.
69-83: Comprehensive coverage of filesystem buffering settings.The explanations of
storage.pause_on_chunks_overlimitandstorage.total_limit_sizeare clear and provide valuable guidance on managing backpressure with filesystem buffering.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
Signed-off-by: Alexa Kreizinger <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@administration/dead-letter-queue.md`:
- Line 60: Conf-format examples use the wrong casing for the logging key;
replace any lowercase or inconsistent occurrences of "log_level" with the Fluent
Bit-style conf key "Log_Level" in all INI/conf examples (e.g., wherever
"log_level: info" or "log_Level" appears) so conf blocks consistently use
"Log_Level" with the correct capitalization and spacing for classic/INI-style
config.
♻️ Duplicate comments (2)
pipeline/buffering.md (2)
94-94: Verify log_level casing consistency between YAML and conf formats.Similar to the issue in
dead-letter-queue.md, the YAML example useslog_level(line 94) while the conf example useslog_Levelwith a capital L (line 116). Confirm whether this casing difference between configuration formats is intentional or if it should be standardized to match the service section documentation.Also applies to: 116-116
7-7: Add alt text to image for accessibility.The
<img>tag is missing thealtattribute, which is required for accessibility. Screen readers and other assistive technologies rely on alt text to convey the image's purpose to users.♿ Proposed fix
-<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=cde12327-09ed-409c-ac02-7c0afa5eff51" /> +<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=cde12327-09ed-409c-ac02-7c0afa5eff51" alt="" />Note: An empty
alt=""attribute is appropriate for tracking pixels that don't convey meaningful content.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
administration/backpressure.mdadministration/configuring-fluent-bit/yaml/service-section.mdadministration/dead-letter-queue.mdpipeline/buffering.md
🧰 Additional context used
🪛 LanguageTool
administration/dead-letter-queue.md
[style] ~43-~43: Consider using a different verb for a more formal wording.
Context: ...play or transform rejected chunks after fixing the underlying issue. - Debugging: ...
(FIX_RESOLVE)
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
|
@esmerel want to take a look? |
Signed-off-by: Alexa Kreizinger <[email protected]>
alexakreizinger
deleted the
alexakreizinger/sc-153843/consolidate-info-from-buffering-backpressure
branch
bear with me, it's a big one. this PR tries to consolidate redundant info by breaking up the "buffering and storage" doc into the existing "buffering" and "backpressure" docs.
in the process I had to rewrite both docs more or less from scratch, although there were a few sections I included pretty much verbatim, and some of my rewrites could still use some work. that'll have to be in a future PR though because after staring at these for two days straight I think I've reached my
mem_buf_limit 🤯I also spun out the dead letter queue info into its own doc since it didn't seem to quite fit in either buffering or backpressure. that one's also a copy/paste job (see above about limits).
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.