fix(loki.source.file): Keep positions for compressed files when reading is finished#5723
fix(loki.source.file): Keep positions for compressed files when reading is finished#5723
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a regression introduced in Alloy 1.8 where compressed files read by loki.source.file would have their position removed from the positions file after being fully consumed (at EOF). This caused the compressed files to be re-ingested on every Alloy restart. The fix adds a shouldKeepPosition() method that preserves the position when decompression is enabled, relying on the existing positions cleanup mechanism to remove entries for files that no longer exist.
Changes:
- Added
shouldKeepPositon()method to the tailer that preserves positions when the component is stopping or when decompression is enabled - Updated
TestTailer_Compressionsto validate that positions are kept after compressed file consumption and that files are not re-ingested on subsequent runs
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
internal/component/loki/source/file/tailer.go |
Adds shouldKeepPositon() method and replaces direct componentStopping() check with it in the stop() method |
internal/component/loki/source/file/tailer_test.go |
Updates TestTailer_Compressions to pass false for componentStopping to verify position retention with decompression |
@thampiotr TestTailer_Compressions was already trying to test this behavior but because it always returned true for componentIsStopping it passed. I changed that to false, this makes this test fail on main but pass in this pr |
Pull Request Details
This is a regression that have been in alloy since 1.8. When tailing compressed files we stop once we hit EOF. The issue is that tailer, and previously decompressor, will remove the stored position because component is not stopping. Then when alloy is re-started we read the files again.
Instead we should never remove the position here when decompression is enabled, position structure will run cleanup on files that no longer exists so it will be able to clean it up when files disapears.
Issue(s) fixed by this Pull Request
Fixes: #5365
Notes to the Reviewer
PR Checklist