Skip to content

Accept the same iterable types supported by zlib, lzma, bz2 modules #79

@tasket

Description

@tasket

Problem

The lack of compatibility with certain buffer types makes this module harder to use and less efficient; one cannot do a simple drop-in replacement of the built-in compression modules.

An in-memory conversion of memoryview() and mmap() to bytes() is first required before zstd can be used on the data; this introduces code complexity (making an exception for zstd) and dramatically increases memory consumption.

>>> import zlib, zstd, mmap
>>>
>>> f   = open("my_file","r+b")
>>> mmf = mmap.mmap(f.fileno(), 0)
>>> mvf = memoryview(mmf)
>>>
>>> c = zlib.compress(mmf)
>>> c = zlib.compress(mvf)
>>>
>>> c = zstd.compress(mmf,3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: argument 1 must be read-only bytes-like object, not mmap.mmap
>>> c = zstd.compress(mvf,3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: argument 1 must be read-only bytes-like object, not memoryview

(Note that converting a memoryview to read-only has no effect here.)

Solution

Support the same buffer types that the Python built-in compression libraries accept.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions