enhance: optimize Storage File Format

[tinswzy]

Currently, Woodpecker writes logs sequentially to files to ensure high throughput and persistence guarantees. However, the current file format either does not utilize compression or loses the ability to efficiently perform partial reads when compression is enabled. This limits performance tuning and restricts future extensibility (e.g., range-based recovery, fast seeking).

The goal of this issue is to design and implement a new storage file format that supports both compression and efficient partial reads.
Goals:

• Support compression: Reduce storage footprint and improve I/O efficiency.
• Enable partial reads: Allow reading specific data blocks without decompressing the entire file.
• Preserve sequential write performance: Writing must remain high-throughput and append-friendly.
• Ensure extensibility: Format should be designed to support future metadata like checksums, versioning, and block indices.

Technical Suggestions:

• Use block-based compression (e.g., compress every N records as a block).
• Record metadata for each block: offset, length, compression type.
• Add a lightweight block index section to enable fast seeks.
• Consider compression algorithms like Snappy or Zstd for a balance of speed and ratio.

Any better solution is also acceptable

[Wenrh2004]

I'm interested in this part. Is there anything I can do to help?

[tinswzy]

I'm interested in this part. Is there anything I can do to help?

Sure, you can take a look at the server/storage/codec code. It features a unified append file format for multiple modes. You can reserve flags to define compression and encryption types. For reading and writing, use the corresponding flags for compression and decompression.
Selecting a more appropriate algorithm based on this format to implement and evaluate the benefits. Very glad and looking forward to your contribution.

[xiaofan-luan]

@tinswzy
do we support zstd compression as our format?

what else is left in this pr?

[tinswzy]

do we support zstd compression as our format?

Yes, the current format can support compression extensions.
The format reserves the flag as the compression type, and the data is read and written at the block level, allowing the block data part to be compressed data. Through the block header, it is possible to quickly filter whether the data within this block needs to be decompressed.