Skip to content

Conversation

@jpinsonneau
Copy link
Contributor

What this PR does / why we need it:

Fixes a file descriptor leak in the TSDB compactor's setupBuilder function that occurs when processing many source index files. The leak causes file descriptors to accumulate during compaction, eventually leading to resource exhaustion and "no space left on device" errors.

Which issue(s) this PR fixes:
Fixes #19514

Special notes for your reviewer:

Created a dedicated test case for that issue.

  • Before fix:
go test -v ./pkg/storage/stores/shipper/indexshipper/tsdb -run TestSetupBuilder_FileDescriptorLeak
=== RUN   TestSetupBuilder_FileDescriptorLeak
    compactor_fd_leak_test.go:148: File descriptor stats: initial=8, peak=58, avg=35, final=8, increase=50, numFiles=50, samples=2
    compactor_fd_leak_test.go:164: File descriptor leak detected! Peak increase: 50 (expected leak: ~50, allowed: 20). This indicates files are not being closed immediately during processing. With the fix, file descriptors should be closed right after use, not deferred until function return. FD samples show accumulation pattern.
--- FAIL: TestSetupBuilder_FileDescriptorLeak (0.24s)
FAIL
FAIL	github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb	0.264s
FAIL
  • After fix:
go test -v ./pkg/storage/stores/shipper/indexshipper/tsdb -run TestSetupBuilder_FileDescriptorLeak
=== RUN   TestSetupBuilder_FileDescriptorLeak
    compactor_fd_leak_test.go:148: File descriptor stats: initial=8, peak=11, avg=10, final=8, increase=3, numFiles=50, samples=2
--- PASS: TestSetupBuilder_FileDescriptorLeak (0.48s)
PASS
ok  	github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb	0.518s

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@jpinsonneau jpinsonneau requested a review from a team as a code owner December 2, 2025 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

File descriptor leak in compactor (Loki 3.3.2, backend pod constantly increasing open REG files in tsdb-shipper-cache)

1 participant