-
Notifications
You must be signed in to change notification settings - Fork 30
Open
Labels
Description
Problem
The lack of compatibility with certain buffer types makes this module harder to use and less efficient; one cannot do a simple drop-in replacement of the built-in compression modules.
An in-memory conversion of memoryview() and mmap() to bytes() is first required before zstd can be used on the data; this introduces code complexity (making an exception for zstd) and dramatically increases memory consumption.
>>> import zlib, zstd, mmap
>>>
>>> f = open("my_file","r+b")
>>> mmf = mmap.mmap(f.fileno(), 0)
>>> mvf = memoryview(mmf)
>>>
>>> c = zlib.compress(mmf)
>>> c = zlib.compress(mvf)
>>>
>>> c = zstd.compress(mmf,3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 1 must be read-only bytes-like object, not mmap.mmap
>>> c = zstd.compress(mvf,3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 1 must be read-only bytes-like object, not memoryview
(Note that converting a memoryview to read-only has no effect here.)
Solution
Support the same buffer types that the Python built-in compression libraries accept.