Skip to content

fix(loki.source.file): Keep positions for compressed files when reading is finished#5723

Open
kalleep wants to merge 4 commits intomainfrom
kalleep/loki-source-file-decompression-position
Open

fix(loki.source.file): Keep positions for compressed files when reading is finished#5723
kalleep wants to merge 4 commits intomainfrom
kalleep/loki-source-file-decompression-position

Conversation

@kalleep
Copy link
Contributor

@kalleep kalleep commented Mar 9, 2026

Pull Request Details

This is a regression that have been in alloy since 1.8. When tailing compressed files we stop once we hit EOF. The issue is that tailer, and previously decompressor, will remove the stored position because component is not stopping. Then when alloy is re-started we read the files again.

Instead we should never remove the position here when decompression is enabled, position structure will run cleanup on files that no longer exists so it will be able to clean it up when files disapears.

Issue(s) fixed by this Pull Request

Fixes: #5365

Notes to the Reviewer

PR Checklist

  • Documentation added
  • Tests updated
  • Config converters updated

@kalleep kalleep requested a review from a team as a code owner March 9, 2026 13:30
@kalleep kalleep added the backport/v1.14 Backport to release/v1.14 label Mar 9, 2026
@kalleep kalleep requested a review from Copilot March 9, 2026 13:35
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a regression introduced in Alloy 1.8 where compressed files read by loki.source.file would have their position removed from the positions file after being fully consumed (at EOF). This caused the compressed files to be re-ingested on every Alloy restart. The fix adds a shouldKeepPosition() method that preserves the position when decompression is enabled, relying on the existing positions cleanup mechanism to remove entries for files that no longer exist.

Changes:

  • Added shouldKeepPositon() method to the tailer that preserves positions when the component is stopping or when decompression is enabled
  • Updated TestTailer_Compressions to validate that positions are kept after compressed file consumption and that files are not re-ingested on subsequent runs

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
internal/component/loki/source/file/tailer.go Adds shouldKeepPositon() method and replaces direct componentStopping() check with it in the stop() method
internal/component/loki/source/file/tailer_test.go Updates TestTailer_Compressions to pass false for componentStopping to verify position retention with decompression

Copy link
Contributor

@thampiotr thampiotr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A test is missing

@kalleep
Copy link
Contributor Author

kalleep commented Mar 9, 2026

A test is missing

@thampiotr TestTailer_Compressions was already trying to test this behavior but because it always returned true for componentIsStopping it passed.

I changed that to false, this makes this test fail on main but pass in this pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport/v1.14 Backport to release/v1.14

Projects

None yet

Development

Successfully merging this pull request may close these issues.

loki: loki.source.file will read already consumed compressed files on restarts.

3 participants