Skip to content

Conversation

@zanieb
Copy link
Member

@zanieb zanieb commented Aug 16, 2025

I verified this on macOS

❯ python -c "from compression import zstd; print(zstd.CompressionParameter.nb_workers.bounds())"
(0, 256)

See #761

I'm not sure if this is all of the performance difference, but using threads should make a significant difference. #761 (comment) notes that threads are not used by default, so I think there's more going on — but we should enable this at build time regardless. See also #768.

@zanieb zanieb added platform:darwin Specific to the macOS platform platform:linux Specific to the Linux platform python:3.14 labels Aug 16, 2025
@zanieb zanieb added the performance Potential performance improvement label Aug 16, 2025
@zanieb
Copy link
Member Author

zanieb commented Aug 16, 2025

We can add a test case that verifies this at runtime.

@Rogdham
Copy link

Rogdham commented Aug 16, 2025

This does not solve #761 (I'm having the same timings as on the issue), however it is a good thing that should be done anyways.

I can confirm that this PR adds support for multithreading.

For example, without this PR, the following exception would be raised:

>>> from compression import zstd
>>> zstd.compress(b'foo', options={zstd.CompressionParameter.nb_workers: 2})
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    zstd.compress(b'foo', options={zstd.CompressionParameter.nb_workers: 2})
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/redacted/.local/share/uv/python/cpython-3.14.0rc1-linux-x86_64-gnu/lib/python3.14/compression/zstd/__init__.py", line 148, in compress
    comp = ZstdCompressor(level=level, options=options, zstd_dict=zstd_dict)
ValueError: compression parameter 'nb_workers' received an illegal value 2; the valid range is [0, 0]

Now with this PR (using cpython-3.14.0rc2-x86_64_v3-unknown-linux-gnu-debug-20250816T1313.tar.zst from GitHub action build artefacts):

>>> from compression import zstd
>>> zstd.compress(b'foo', options={zstd.CompressionParameter.nb_workers: 2})
b'(\xb5/\xfd \x03\x19\x00\x00foo'

@zanieb zanieb requested a review from geofft August 18, 2025 12:38
@zanieb zanieb merged commit c744db9 into main Aug 18, 2025
110 checks passed
@zanieb zanieb deleted the zb/zstd-mt branch August 18, 2025 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Potential performance improvement platform:darwin Specific to the macOS platform platform:linux Specific to the Linux platform python:3.14

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants