Merge pull request #44 from pycompression/release_0.5.0

Release 0.5.0

Author: rhpvorderman

.github/workflows/ci.yml

@@ -44,6 +44,25 @@ jobs:
         run: tox -e docs
         env:
           PYTHON_ISAL_LINK_DYNAMIC: True
+  mypy:
+    needs: lint
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/[email protected]
+        with:
+          submodules: recursive
+      - name: Set up Python 3.6
+        uses: actions/[email protected]
+        with:
+          python-version: 3.6
+      - name: Install isal
+        run: sudo apt-get install libisal-dev
+      - name: Install tox and upgrade setuptools and pip
+        run: pip install --upgrade tox setuptools pip
+      - name: Mypy checks
+        run: tox -e mypy
+        env:
+          PYTHON_ISAL_LINK_DYNAMIC: True
   twine_check:
     needs: lint
     runs-on: ${{ matrix.os }}

CHANGELOG.rst

@@ -7,6 +7,19 @@ Changelog
 .. This document is user facing. Please word the changes in such a way
 .. that users understand how the changes affect the new version.
 
+version 0.5.0
+-----------------
++ Fix a bug where negative integers were not allowed for the ``adler32`` and
+  ``crc32`` functions in ``isal_zlib``.
++ Provided stubs (type-hint files) for ``isal_zlib`` and ``_isal`` modules.
+  Package is now tested with mypy to ensure correct type information.
++ The command-line interface now reads in blocks of 32K instead of 8K. This
+  improves performance by about 6% when compressing and 11% when decompressing.
+  A hidden ``-b`` flag was added to adjust the buffer size for benchmarks.
++ A ``-c`` or ``--stdout`` flag was added to the CLI interface of isal.igzip.
+  This allows it to behave more like the ``gzip`` or ``pigz`` command line
+  interfaces.
+
 version 0.4.0
 -----------------
 + Move wheel building to cibuildwheel on github actions CI. Wheels are now

README.rst

@@ -43,6 +43,29 @@ a variety of functions to provide zlib/gzip-compatible compression.
 ``igzip`` module which are usable as drop-in replacements for the ``zlib``
 and ``gzip`` modules from the stdlib (with some minor exceptions, see below).
 
+Usage
+-----
+
+Python-isal has faster versions of the stdlib's ``zlib`` and ``gzip`` module
+these are called ``isal_zlib`` and ``igzip`` respectively.
+
+They can be imported as follows
+
+.. code-block:: python
+
+    from isal import isal_zlib
+    from isal import igzip
+
+``isal_zlib`` and ``igzip`` are meant to be used as drop in replacements so
+their api and functions are the same as the stdlib's modules. Except where
+isa-l does not support the same calls as zlib (See differences below).
+
+A full API documentation can be found on `our readthedocs page
+<https://python-isal.readthedocs.io>`_.
+
+``python -m isal.igzip`` implements a simple gzip-like command line
+application (just like ``python -m gzip``).
+
 Installation
 ------------
 Installation with pip
@@ -99,39 +122,13 @@ python-isal is available on conda-forge and can be installed with
 This will automatically install the isa-l library dependency as well, since
 it is available on conda-forge.
 
-
-Usage
------
-
-Python-isal has faster versions of the stdlib's ``zlib`` and ``gzip`` module
-these are called ``isal_zlib`` and ``igzip`` respectively.
-
-They can be imported as follows
-
-.. code-block:: python
-
-    from isal import isal_zlib
-    from isal import igzip
-
-``isal_zlib`` and ``igzip`` were meant to be used as drop in replacements so
-their api and functions are the same as the stdlib's modules. Except where
-isa-l does not support the same calls as zlib (See differences below).
-
-A full API documentation can be found on `our readthedocs page
-<https://python-isal.readthedocs.io>`_.
-
-``python -m isal.igzip`` implements a simple gzip-like command line
-application (just like ``python -m gzip``).
-
 Differences with zlib and gzip modules
 --------------------------------------
 
 + Compression level 0 in ``zlib`` and ``gzip`` means **no compression**, while
   in ``isal_zlib`` and ``igzip`` this is the **lowest compression level**.
   This is a design choice that was inherited from the ISA-L library.
 + Compression levels range from 0 to 3, not 1 to 9.
-+ ``isal_zlib.crc32`` and ``isal_zlib.adler32`` do not support negative
-  numbers for the value parameter.
 + ``zlib.Z_DEFAULT_STRATEGY``, ``zlib.Z_RLE`` etc. are exposed as
   ``isal_zlib.Z_DEFAULT_STRATEGY``, ``isal_zlib.Z_RLE`` etc. for compatibility
   reasons. However, ``isal_zlib`` only supports a default strategy and will
@@ -140,13 +137,14 @@ Differences with zlib and gzip modules
   ``isal_zlib`` supports memory levels smallest, small, medium, large and
   largest. These have been mapped to levels 1, 2-3, 4-6, 7-8 and 9. So
   ``isal_zlib`` can be used with zlib compatible memory levels.
-+ ``isal_zlib`` only supports ``FLUSH``, ``SYNC_FLUSH`` and ``FULL_FLUSH``
-  ``FINISH`` is aliased to ``FULL_FLUSH`` (and works correctly as such).
 + ``isal_zlib`` has a ``compressobj`` and ``decompressobj`` implementation.
   However, the unused_data and unconsumed_tail for the Decompress object, only
   work properly when using gzip compatible compression. (25 <= wbits <= 31).
 + The flush implementation for the Compress object behavious differently from
-  the zlib equivalent.
+  the zlib equivalent. The flush implementation is sufficient for 
+  the ``igzip`` module to work 100% in compliance with the ``gzip`` tests from
+  CPython. It does not however work for all the ``zlib`` compliance tests 
+  (see above). This is an area that still needs work.
 
 Contributing
 ------------

codecov.yml

@@ -0,0 +1,9 @@
+comment: off
+coverage:
+  status:
+    project:
+      default:
+        target: 90  # let's try to hit high standards
+    patch:
+      default:
+        target: 90  # Tests should be written for new features

setup.py

@@ -114,7 +114,7 @@ def build_isa_l():
 
 setup(
     name="isal",
-    version="0.4.0",
+    version="0.5.0",
     description="Faster zlib and gzip compatible compression and "
                 "decompression by providing python bindings for the isa-l "
                 "library.",
@@ -128,7 +128,7 @@ def build_isa_l():
     zip_safe=False,
     packages=find_packages('src'),
     package_dir={'': 'src'},
-    package_data={'isal': ['*.pxd', '*.pyx',
+    package_data={'isal': ['*.pxd', '*.pyx', '*.pyi', 'py.typed',
                            # Include isa-l LICENSE and other relevant files
                            # with the binary distribution.
                            'isa-l/LICENSE', 'isa-l/README.md',

src/isal/__init__.py

@@ -29,4 +29,4 @@
     "__version__"
 ]
 
-__version__ = "0.4.0"
+__version__ = "0.5.0"

src/isal/_isal.pyi

@@ -0,0 +1,24 @@
+# Copyright (c) 2020 Leiden University Medical Center
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+ISAL_MAJOR_VERSION: int
+ISAL_MINOR_VERSION: int
+ISAL_PATCH_VERSION: int
+ISAL_VERSION: str

src/isal/igzip.py

@@ -27,8 +27,6 @@
 import os
 import sys
 
-import _compression
-
 from . import isal_zlib
 
 __all__ = ["IGzipFile", "open", "compress", "decompress", "BadGzipFile"]
@@ -37,10 +35,8 @@
 _COMPRESS_LEVEL_TRADEOFF = isal_zlib.ISAL_DEFAULT_COMPRESSION
 _COMPRESS_LEVEL_BEST = isal_zlib.ISAL_BEST_COMPRESSION
 
-BUFFER_SIZE = _compression.BUFFER_SIZE
-
 try:
-    BadGzipFile = gzip.BadGzipFile
+    BadGzipFile = gzip.BadGzipFile  # type: ignore
 except AttributeError:  # Versions lower than 3.8 do not have BadGzipFile
     BadGzipFile = OSError
 
@@ -80,7 +76,8 @@ def open(filename, mode="rb", compresslevel=_COMPRESS_LEVEL_TRADEOFF,
             raise ValueError("Argument 'newline' not supported in binary mode")
 
     gz_mode = mode.replace("t", "")
-    if isinstance(filename, (str, bytes, os.PathLike)):
+    # __fspath__ method is os.PathLike
+    if isinstance(filename, (str, bytes)) or hasattr(filename, "__fspath__"):
         binary_file = IGzipFile(filename, gz_mode, compresslevel)
     elif hasattr(filename, "read") or hasattr(filename, "write"):
         binary_file = IGzipFile(None, gz_mode, compresslevel, filename)
@@ -276,33 +273,46 @@ def main():
         "-d", "--decompress", action="store_false",
         dest="compress",
         help="Decompress the file instead of compressing.")
+    parser.add_argument("-c", "--stdout", action="store_true",
+                        help="write on standard output")
+    # -b flag not taken by either gzip or igzip. Hidden attribute. Above 32K
+    # diminishing returns hit. _compression.BUFFER_SIZE = 8k. But 32K is about
+    # ~6% faster.
+    parser.add_argument("-b", "--buffer-size",
+                        default=32 * 1024, type=int,
+                        help=argparse.SUPPRESS)
     args = parser.parse_args()
 
     compresslevel = args.compresslevel or _COMPRESS_LEVEL_TRADEOFF
 
-    if args.file is None:
-        if args.compress:
-            in_file = sys.stdin.buffer
-            out_file = IGzipFile(mode="wb", compresslevel=compresslevel,
-                                 fileobj=sys.stdout.buffer)
-        else:
-            in_file = IGzipFile(mode="rb", fileobj=sys.stdin.buffer)
-            out_file = sys.stdout.buffer
-    else:
-        if args.compress:
-            in_file = io.open(args.file, mode="rb")
-            out_file = open(args.file + ".gz", mode="wb",
-                            compresslevel=compresslevel)
-        else:
-            base, extension = os.path.splitext(args.file)
-            if extension != ".gz":
-                print(f"filename doesn't end in .gz: {args.file}")
-                return
-            in_file = open(args.file, "rb")
-            out_file = io.open(base, "wb")
+    # Determine input file
+    if args.compress and args.file is None:
+        in_file = sys.stdin.buffer
+    elif args.compress and args.file is not None:
+        in_file = io.open(args.file, mode="rb")
+    elif not args.compress and args.file is None:
+        in_file = IGzipFile(mode="rb", fileobj=sys.stdin.buffer)
+    elif not args.compress and args.file is not None:
+        base, extension = os.path.splitext(args.file)
+        if extension != ".gz":
+            raise ValueError(f"filename doesn't end in .gz: {args.file}. ")
+        in_file = open(args.file, "rb")
+
+    # Determine output file
+    if args.compress and (args.file is None or args.stdout):
+        out_file = IGzipFile(mode="wb", compresslevel=compresslevel,
+                             fileobj=sys.stdout.buffer)
+    elif args.compress and args.file is not None:
+        out_file = open(args.file + ".gz", mode="wb",
+                        compresslevel=compresslevel)
+    elif not args.compress and (args.file is None or args.stdout):
+        out_file = sys.stdout.buffer
+    elif not args.compress and args.file is not None:
+        out_file = io.open(base, "wb")
+
     try:
         while True:
-            block = in_file.read(BUFFER_SIZE)
+            block = in_file.read(args.buffer_size)
             if block == b"":
                 break
             out_file.write(block)

src/isal/isal_zlib.pyi

@@ -0,0 +1,75 @@
+# Copyright (c) 2020 Leiden University Medical Center
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+ISAL_BEST_SPEED: int
+ISAL_BEST_COMPRESSION: int
+ISAL_DEFAULT_COMPRESSION: int
+Z_BEST_SPEED: int
+Z_BEST_COMPRESSION: int
+Z_DEFAULT_COMPRESSION: int
+
+DEF_BUF_SIZE: int
+DEF_MEM_LEVEL: int
+MAX_WBITS: int
+ISAL_DEFAULT_HIST_BITS: int
+
+DEFLATED: int
+
+Z_DEFAULT_STRATEGY: int
+Z_RLE: int
+Z_HUFFMAN_ONLY: int
+Z_FILTERED: int
+Z_FIXED: int
+
+ISAL_NO_FLUSH: int
+ISAL_SYNC_FLUSH: int
+ISAL_FULL_FLUSH: int
+
+Z_NO_FLUSH: int
+Z_SYNC_FLUSH: int
+Z_FINISH: int
+
+class IsalError(OSError): ...
+
+error: IsalError
+
+def adler32(data, value: int = ...) -> int: ...
+def crc32(data, value: int = ...) -> int: ...
+
+def compress(data, level: int = ..., wbits: int = ...) -> bytes: ...
+def decompress(data, wbits: int = ..., bufsize: int = ...) -> bytes: ...
+
+class Compress:
+    def compress(self, data) -> bytes: ...
+    def flush(self, mode: int = ...) -> bytes: ...
+
+class Decompress:
+    unused_data: bytes
+    unconsumed_tail: bytes
+    eof: bool
+    crc: int
+
+    def decompress(self, data, max_length: int = ...) -> bytes: ...
+    def flush(self, length: int = ...) -> bytes: ...
+
+def compressobj(level: int = ..., method: int = ..., wbits: int = ...,
+                memLevel: int = ..., strategy: int = ..., zdict = ...
+                ) -> Compress: ...
+def decompressobj(wbits: int = ..., zdict = ...) -> Decompress: ...