zstd 1.5.7 #208334
Merged
zstd 1.5.7 #208334
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
bump-formula-pr
bevanjkay
approved these changes
Feb 19, 2025
chenrui333
added
the
CI-skip-recursive-dependents
daeho-ro
approved these changes
Feb 20, 2025
Contributor
|
🤖 An automated task has requested bottles to be published to this PR. |
github-actions
bot
added
the
CI-published-bottle-commits
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by
brew bumpCreated with
brew bump-formula-pr.Details
release notes
These improvements are particularly beneficial in data centers and databases, where data is often divided into small segments to enhance seekability. A notable example is RocksDB, which is frequently configured with block sizes around 16 KB. Such systems will experience an immediate performance boost by updating their compression library to
v1.5.7.Additionally, the speed improvements partially extend to dictionary compression, with observed gains ranging from none to +15%.
Substantial
--patch-fromimprovementsThe
--patch-fromfunctionality of thezstdCLI allows for differential compression of a file given an older version, similar tobsdiff. While compression levels impact processing speed and compression ratio as expected, at high compression levels (18+), speed was previously reduced to uncomfortable levels.v1.5.7largely mitigates the speed impact of high compression levels 18+, as illustrated in the following scenario, which compresseslinux-6.13.taras a difference fromlinux-6.12.tarusing 6 threads:Additionally, compression ratio of
--patch-fromis also substantially improved across the board, including higher compression levels:--maxCompression ratio improvements for large files
The compression ratio has been enhanced slightly for large data across all compression levels, thanks to a refined approach in selecting block boundaries. This improvement is more pronounced at lower compression levels (≤ 15), where the previous boundary selection method was relatively basic. In contrast, higher compression levels (16+) already employed some analysis, but still benefit from the introduction of the new heuristic, resulting in marginally better compression ratios.
A comparison of the compression ratios between
v1.5.6andv1.5.7is presented below:v1.5.6v1.5.7Command-Line Interface Updates
Recognizing the prevalence of multi-core systems,
zstdnow employs multiple threads by default, thanks to #4211 by @daniellerozenblitIn
v1.5.7, systems will see default threading scaled according to capability, but remains cautious to mitigate risks of overloading surprises on existing user base, with a maximum of 4 threads by default.For example, a typical "low-end" system with 4 cores and hyperthreading will employ 2 threads by default. A "higher-end" system with 8 cores and hyperthreading will employ 4 threads by default.
These modifications only impact default settings. As usual, the number of threads that
zstduses can also be set manually, either via command line or via environment variables.Additionally, a new
zstdcommand--maxis available, designed to maximize compression ratios by leveraging all parameters, at the expense of time and memory. It's slower and more resource hungry than--ultra -22, but achieves denser compressed archives if that's the goal. For instance, compressingenwik9(1,000,000,000 bytes) at--maxlevel on a i7-9700k platform yields:--ultra -22--maxBug Fixes
v1.5.7resolves a long standing and very rare compression issue in 32-bit mode, which can be triggered during long-lasting sessions (sameZSTD_CCtx*being continuously reused). This issue required several conditions to be met and a large amount of data to occur, but at the scale ofzstdusage, it has been observed and reported (#4292).In summary, if you use
libzstdin 32-bit mode, this update is highly recommended.Miscellaneous
The release also brings its lot of various improvements, impacting build scripts (cmake, meson, Apple Framework, Visual Studio), documentation and portability (QNX, FreeBSD, Hurd, icc/icx).
Notably, there are measurable performance gains for binaries produced with Visual Studio, as a cumulative result of several contributions.
Changelog:
--patch-fromat high compression levels (#4276)--patch-fromcompression ratios, notably at high levels (#4288)dfast, aka levels 3 and 4 (#4171)--maxcommand (#4290)ZSTD_compressSequencesAndLiterals()(#4217, #4232)ZSTD_getFrameHeader()now works on skippable frames (#4228)msbuildversion autodetection, support VS2022, by @ManuelBlancmesonbuild by @artem and @Victor-C-Zhang, and on Windows by @bgilbertNew Contributors
-pthreadto static linking flags on Windows facebook/zstd#3931Full Changelog: facebook/zstd@v1.5.6...v1.5.7