Allow Zstandard to decompress multiple concatenated frames

[mkitti]

Currently, numcodecs assumes that a buffer, a pointer and a size, containing Zstandard compressed only consists of a single frame. However, this is not necessarily the case in that a single buffer may contain multiple frames. A buffer with multiple compressed frames is easily constructed by simply concatenating two encoded buffers. Currently, when decoding this buffer numcodecs.Zstd will report that the destination buffer is too small.

In [1]: import numcodecs

In [2]: codec = numcodecs.Zstd()

In [3]: hello_world = codec.encode(b"Hello world!")

In [4]: codec.decode(hello_world*2)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[4], line 1
----> 1 codec.decode(hello_world*2)

File numcodecs/zstd.pyx:261, in numcodecs.zstd.Zstd.decode()

File numcodecs/zstd.pyx:221, in numcodecs.zstd.decompress()

RuntimeError: Zstd decompression error: b'Destination buffer is too small'

It will even do this when providing an output buffer of sufficient size.

In [5]: dest_buffer = bytearray(len(b"Hello world!")*2)

In [6]: codec.decode(hello_world*2, out=dest_buffer)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[6], line 1
----> 1 codec.decode(hello_world*2, out=dest_buffer)

File numcodecs/zstd.pyx:261, in numcodecs.zstd.Zstd.decode()

File numcodecs/zstd.pyx:221, in numcodecs.zstd.decompress()

RuntimeError: Zstd decompression error: b'Destination buffer is too small'

The expectation is that the decompressed output should simply the concatenation of the original uncompressed buffers.

In [7]: import pyzstd

In [8]: pyzstd.decompress(hello_world*2)
Out[8]: b'Hello world!Hello world!'

The flaw is that numcodecs.Zstd currently runs ZSTD_getFrameContentSize only once to get the uncompressed size of the first frame. It does not consider whether there may be other frames in the source buffer.

This pull request introduces a Cython function findTotalContentSize which iterates through all the frames in the buffer, gets the sizes of the uncompressed data from each frame, and then returns the sum. This allows numcodecs.Zstd to decompress the entire buffer.

>>> import numcodecs
>>> codec = numcodecs.Zstd()
>>> hello_world = codec.encode(b"Hello World!")
>>> codec.decode(hello_world*2)
b'Hello World!Hello World!'
>>> out = bytearray(len("Hello World!")*2)
>>> codec.decode(hello_world*2, out=out)
bytearray(b'Hello World!Hello World!')
>>> out
bytearray(b'Hello World!Hello World!')

One application of this is that a buffer containing multiple chunks of a Zarr v3 shard contiguously could be decompressed in a single step.

• Add support for multiple zstd frames in decompression
• Add release notes

[Description of PR]

TODO:

• Unit tests and/or doctests in docstrings
• Tests pass locally
• Docstrings and API docs for any new/modified user-facing classes and functions
• Changes documented in docs/release.rst
• Docs build locally
• GitHub Actions CI passes
• Test coverage to 100% (Codecov passes)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (f42a233) to head (4766a06).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #757   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           64        64           
  Lines         2804      2824   +20     
=========================================
+ Hits          2804      2824   +20     

Files with missing lines	Coverage Δ
numcodecs/tests/test_pyzstd.py	100.00% <ø> (ø)
numcodecs/tests/test_zstd.py	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mkitti]

After this pull request

>>> from numcodecs import Zstd
>>> codec = Zstd()
>>> bytes = codec.encode(b"Hello ")
>>> codec.decode(bytes)
b'Hello '
>>> codec.decode(bytes+bytes)
b'Hello Hello '