out_file: add logs rotation feature #11352
Conversation
Signed-off-by: SagiROosto <[email protected]>
Signed-off-by: SagiROosto <[email protected]>
📝 WalkthroughWalkthroughThe pull request adds comprehensive file rotation and optional gzip compression capabilities to the out_file plugin. Implementation includes rotation-aware flush handling with per-file size tracking, configurable retention policies, streaming gzip compression pipeline, cross-platform directory creation, thread-safe per-entry locking, and an extensive test suite covering rotation scenarios, multiple output formats, and multithreaded workloads. Changes
Sequence DiagramssequenceDiagram
participant Fluent Bit
participant Flush Handler
participant Size Tracker
participant Rotation
participant Gzip
participant Filesystem
Fluent Bit->>Flush Handler: cb_file_flush(records)
Flush Handler->>Size Tracker: find_or_create_file_size_entry(filename)
Size Tracker->>Size Tracker: lock entry
Flush Handler->>Filesystem: open file (mkdir if needed)
Flush Handler->>Filesystem: write formatted records
Flush Handler->>Size Tracker: update size
alt size exceeds max_size and rotation enabled
Flush Handler->>Rotation: rotate_file(ctx, filename)
Rotation->>Filesystem: rename to timestamped name
alt gzip enabled
Rotation->>Gzip: gzip_compress_file(timestamped, .gz)
Gzip->>Gzip: write_gzip_header()
loop chunks
Gzip->>Gzip: deflate compress chunk
Gzip->>Gzip: update crc
end
Gzip->>Gzip: write_gzip_footer(crc, size)
Gzip->>Filesystem: write compressed data
Filesystem->>Filesystem: delete original
end
Rotation->>Rotation: cleanup_old_files(ctx, max_files)
Rotation->>Filesystem: remove oldest rotated files
Flush Handler->>Size Tracker: reset size counter
end
Size Tracker->>Size Tracker: unlock entry
Flush Handler->>Fluent Bit: return status
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~75 minutes Multiple interrelated systems require careful review: streaming gzip implementation with CRC/header/footer handling, per-file locking and thread-safety guarantees, rotation state management with size tracking, cross-platform directory creation, error handling across file operations, and comprehensive test scenarios validating all code paths and format combinations. Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 628922c6a1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @plugins/out_file/file.c:
- Around line 222-238: The error paths after flb_lock_init(&ctx->list_lock) do
not destroy the lock before freeing ctx; update the failure branches for
flb_output_config_map_set and the ctx->max_files validation to call
flb_lock_destroy(&ctx->list_lock) (or the appropriate destroy function) prior to
flb_free(ctx) and returning -1 so that the lock is cleaned up; locate these
spots around the mk_list_init, flb_lock_init, flb_output_config_map_set, and
flb_plg_error calls and add the destroy call in both error branches.
🧹 Nitpick comments (2)
tests/runtime/out_file_logrotate.c (2)
1040-1041: Array oversized for declared usage.
pthread_t threads[8]is declared but onlynum_threads = 4threads are used. Consider sizing the array to match actual usage or using a constant.♻️ Suggested fix
- pthread_t threads[8]; - struct thread_data thread_data[8]; + pthread_t threads[4]; + struct thread_data thread_data[4];
1160-1164: Remove unused variable.The
timestampvariable is generated but never used in this test function.♻️ Suggested fix
- time_t now = time(NULL); - struct tm *tm_info = localtime(&now); - char timestamp[32]; - - strftime(timestamp, sizeof(timestamp), "%Y%m%d_%H%M%S", tm_info);
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
plugins/out_file/file.ctests/runtime/CMakeLists.txttests/runtime/out_file_logrotate.c
🧰 Additional context used
🧠 Learnings (5)
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
tests/runtime/CMakeLists.txt
📚 Learning: 2025-08-29T06:25:27.250Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.
Applied to files:
plugins/out_file/file.c
📚 Learning: 2025-08-29T06:25:02.561Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.
Applied to files:
plugins/out_file/file.c
📚 Learning: 2025-08-29T06:24:26.170Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.
Applied to files:
plugins/out_file/file.c
📚 Learning: 2025-08-29T06:24:55.855Z
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.
Applied to files:
plugins/out_file/file.c
🧬 Code graph analysis (2)
tests/runtime/out_file_logrotate.c (3)
include/fluent-bit/flb_mem.h (1)
flb_free(126-128)src/flb_lib.c (9)
flb_create(143-225)flb_service_set(652-678)flb_input(266-276)flb_output(279-289)flb_output_set(520-551)flb_start(983-994)flb_lib_push(843-870)flb_stop(1011-1055)flb_destroy(228-263)plugins/in_head/in_head.c (1)
read_bytes(87-106)
plugins/out_file/file.c (6)
include/fluent-bit/flb_mem.h (1)
flb_calloc(84-96)src/flb_lock.c (4)
flb_lock_init(6-17)flb_lock_acquire(32-65)flb_lock_release(67-100)flb_lock_destroy(19-30)include/fluent-bit/flb_output.h (1)
flb_output_config_map_set(1306-1333)src/flb_pack.c (1)
flb_msgpack_to_json_str(1459-1500)src/flb_sds.c (2)
flb_sds_create(78-90)flb_sds_destroy(389-399)src/flb_log_event_decoder.c (2)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)
🪛 ast-grep (0.40.3)
tests/runtime/out_file_logrotate.c
[warning] 586-586: A check is done with stat and then the file is used. There is no guarantee that the status of the file has not changed since the call to stat which may allow attackers to bypass permission checks.
Context: fopen
Note: [CWE-367]: Time-of-check Time-of-use (TOCTOU) Race Condition [REFERENCES]
- https://wiki.sei.cmu.edu/confluence/display/c/FIO45-C.+Avoid+TOCTOU+race+conditions+while+accessing+files
(file-stat-before-action-c)
🔇 Additional comments (20)
tests/runtime/out_file_logrotate.c (9)
1-34: Well-structured test file with comprehensive coverage.Good job setting up cross-platform support with appropriate macros for Windows/POSIX and including the necessary test data. The test function declarations provide clear visibility into the test coverage.
80-203: Robust cross-platform directory cleanup implementation.The helper properly handles both Windows and POSIX platforms, skips special entries, and continues cleanup even on partial failures.
206-248: LGTM!The file counting helper is correctly implemented for both platforms.
254-315: LGTM!The polling helper is well-implemented with proper resource cleanup on each iteration.
318-350: LGTM!The file reading helper properly handles error cases and memory cleanup. The TOCTOU concern (stat before fopen) is acceptable for test code where files are under the test's control.
353-670: Comprehensive format tests with consistent structure.All format tests follow a good pattern: setup, configure, push data, verify content, cleanup. The assertions check format-specific characteristics (CSV commas, LTSV colons, plain JSON, msgpack binary, template substitution).
1358-1403: Good negative test for configuration validation.Testing that
flb_startfails with invalidmax_filesvalues (0 and -1) properly validates the configuration checks in the plugin init.
1405-1483: Excellent edge case coverage for gzip compression.Testing the exact 64KB chunk boundary ensures the streaming gzip implementation handles the edge case where file size equals the chunk size.
1101-1102: Thewait_for_file()function is correctly available. It is defined as a static inline function intests/runtime/flb_tests_runtime.h.in(lines 29-32) and accessible through the#include "flb_tests_runtime.h"header at line 3 of the file. The function call at line 1101 is valid.plugins/out_file/file.c (10)
65-75: Well-designed per-file tracking structure.The
file_file_sizestructure with per-entry locks enables fine-grained concurrent access to different output files.
118-200: Good security measure for tag sanitization.The function properly prevents path traversal attacks by collapsing ".." and "." segments, sanitizing unsafe characters, and handling edge cases like empty tags.
706-880: Solid streaming gzip implementation.The function correctly handles:
- Streaming compression with bounded memory usage
- Proper gzip header/footer with CRC32
- The edge case where file size is exactly a multiple of chunk size (lines 819-843)
- Resource cleanup on all paths
882-969: Well-structured rotation logic with proper cleanup.The function properly handles the rotation workflow: rename, optional gzip compression, cleanup of failed gzip, and size reset. The lock contract is clearly documented.
1155-1173: Bubble sort is adequate for expected file counts.The O(n²) sorting algorithm is acceptable here since
max_filesis typically small (default 7). For very large retention counts, consider using qsort.
1262-1296: Correct hand-over-hand locking pattern.The locking sequence is properly implemented:
- Acquire list_lock
- Find/create entry
- Acquire entry->lock
- Release list_lock
- Perform file operations under entry->lock
- Release entry->lock
This allows concurrent operations on different files while preventing races on the same file.
1562-1570: LGTM!The cleanup section properly frees allocated memory for
directoryandbase_filenameon all exit paths.
1572-1601: Proper cleanup of all resources.The exit handler correctly:
- Acquires list_lock
- Iterates and destroys each entry's lock and filename
- Releases and destroys list_lock
- Frees the context
1639-1654: Good configuration defaults.The rotation feature is disabled by default (
files_rotation = false) preserving backward compatibility. The defaults (100MB max size, 7 files, gzip enabled) are sensible for typical log rotation scenarios.
1658-1668: Plugin configuration is appropriate.With
.workers = 1, the plugin uses a single worker thread. The per-file locking still provides value when the same plugin instance writes to multiple files (based on tag).tests/runtime/CMakeLists.txt (1)
235-238: Correctly integrated into the build system.The new test file is properly added to the Linux-only test block using the existing
FLB_OUT_FILEflag, ensuring it builds alongside the existing out_file tests.
1 participant
This PR present a new rotation feature to
out_file. This will answer #7572 question and solve #7541 suggestion.This feature is highly required in bare-metal or physical machines where you have finite and limited storage.
The suggested code disables the rotation feature by default to keep backward compatibility without affecting existing plugin users.
(this replaces closed #11110 PR)
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
Release Notes
New Features
Tests
✏️ Tip: You can customize this high-level summary in your review settings.