Skip to content

Conversation

sekiyama58
Copy link
Contributor

@sekiyama58 sekiyama58 commented Jan 24, 2025

Currently the dictionary is setup every time when it is passed to compress(dict:) / decompress(dict:), but the dictionary setup is costly process.
When the dictionary is repeatedly used, creating CDict / DDict in advance makes the compression / decompression much efficient.
This adds the Zstd::CDict & Zstd::DDict class to manage pre-loaded dictionary references.

Benchmark

CDict makes it x~13 times faster to compress spec/user_springmt.json 100 times with spec/dictionary.

                               user     system      total        real
compress with String dict  0.012185   0.001020   0.013205 (  0.013205)
compress with CDict        0.000927   0.000040   0.000967 (  0.000973)
require 'bundler/setup'
require 'zstd-ruby'
require 'benchmark'

data = IO.read("spec/user_springmt.json")
dict = IO.read("spec/dictionary")
cdict = Zstd::CDict.new(dict, 6)
ddict = Zstd::DDict.new(dict)

Benchmark.bm 24 do |x|
  x.report("compress with String dict") do
    100.times { Zstd.compress(data, dict: dict) }
  end
  x.report("compress with CDict") do
    100.times { Zstd.compress(data, dict: cdict) }
  end
end

@SpringMT SpringMT merged commit 116ee47 into SpringMT:main Mar 7, 2025
6 checks passed
@SpringMT
Copy link
Owner

SpringMT commented Mar 7, 2025

Thx!

@catlee
Copy link
Contributor

catlee commented Jul 4, 2025

Is there any chance we can get a release with this support in it?

@SpringMT
Copy link
Owner

SpringMT commented Jul 6, 2025

I released v1.5.6.7 🙇 https://rubygems.org/gems/zstd-ruby/versions/1.5.6.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants