Skip to content

Read lz4-compressed files and remove after reading#729

Merged
DmitryRomanov merged 20 commits intomasterfrom
lz4-file-read
Dec 29, 2025
Merged

Read lz4-compressed files and remove after reading#729
DmitryRomanov merged 20 commits intomasterfrom
lz4-file-read

Conversation

@DmitryRomanov
Copy link
Copy Markdown
Collaborator

@DmitryRomanov DmitryRomanov commented Dec 26, 2024

  • read compressed lz4 files (defines by extension)
  • add option remove_after to file input plugin - After reaching EOF, the number of seconds to wait before removing the file, unless new data is written

@DmitryRomanov DmitryRomanov force-pushed the lz4-file-read branch 5 times, most recently from 3ccae83 to 6ecb2c9 Compare February 11, 2025 10:24
@DmitryRomanov DmitryRomanov force-pushed the lz4-file-read branch 2 times, most recently from 88a86c0 to a2b978a Compare February 14, 2025 08:30
@DmitryRomanov DmitryRomanov force-pushed the lz4-file-read branch 2 times, most recently from 0da1f9f to e630f5c Compare November 25, 2025 15:57
@DmitryRomanov DmitryRomanov changed the title lz4 decompress in file worker Read lz4-compressed files and remove after reading Nov 27, 2025
@DmitryRomanov DmitryRomanov marked this pull request as ready for review November 27, 2025 08:49
@DmitryRomanov DmitryRomanov force-pushed the lz4-file-read branch 3 times, most recently from 76565cd to a940949 Compare December 25, 2025 12:39
@DmitryRomanov
Copy link
Copy Markdown
Collaborator Author

DmitryRomanov commented Dec 29, 2025

Benches

before (file.d v0.73.1)

go run ./bench/server
info	filebeat 50
info	started, waiting for first request
info	stats bytes=301Mb, requests=405, throughput: 302.67Mb/s
info	stats bytes=503Mb, requests=675, throughput: 252.73Mb/s
info	stats bytes=728Mb, requests=981, throughput: 242.69Mb/s
info	stats bytes=1044Mb, requests=1409, throughput: 260.88Mb/s
info	stats bytes=1309Mb, requests=1765, throughput: 261.98Mb/s
info	stats bytes=1605Mb, requests=2162, throughput: 267.71Mb/s
info	stats bytes=1872Mb, requests=2521, throughput: 267.31Mb/s
info	stats bytes=2186Mb, requests=2942, throughput: 273.08Mb/s
info	stats bytes=2434Mb, requests=3280, throughput: 270.53Mb/s
info	final result: 266.30Mb/s
goos: linux
goarch: amd64
pkg: github.com/ozontech/file.d/plugin/input/file
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkLightJsonReadPar-4   	       3	2277086821 ns/op	 637.32 MB/s	200504917 B/op	 2221101 allocs/op
PASS
ok  	github.com/ozontech/file.d/plugin/input/file	12.499s

after

go run ./bench/server
info	filebeat 50
info	started, waiting for first request
info	stats bytes=276Mb, requests=369, throughput: 289.29Mb/s
info	stats bytes=518Mb, requests=695, throughput: 259.48Mb/s
info	stats bytes=809Mb, requests=1091, throughput: 270.03Mb/s
info	stats bytes=1028Mb, requests=1390, throughput: 257.19Mb/s
info	stats bytes=1327Mb, requests=1787, throughput: 266.75Mb/s
info	stats bytes=1593Mb, requests=2146, throughput: 265.78Mb/s
info	stats bytes=1920Mb, requests=2585, throughput: 274.27Mb/s
info	stats bytes=2244Mb, requests=3018, throughput: 280.38Mb/s
info	stats bytes=2568Mb, requests=3462, throughput: 285.32Mb/s
info	final result: 287.85Mb/s
goos: linux
goarch: amd64
pkg: github.com/ozontech/file.d/plugin/input/file
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkLightJsonReadPar-4   	       3	2166702207 ns/op	 669.79 MB/s	192067794 B/op	 2122678 allocs/op
PASS
ok  	github.com/ozontech/file.d/plugin/input/file	11.546s

@DmitryRomanov DmitryRomanov merged commit e4555ae into master Dec 29, 2025
15 of 16 checks passed
@DmitryRomanov DmitryRomanov deleted the lz4-file-read branch December 29, 2025 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants