Merge pull request #73 from pycompression/igzip_lib2

Add an igzip_lib module.

Author: rhpvorderman

.github/workflows/ci.yml

@@ -197,7 +197,9 @@ jobs:
           CIBW_TEST_REQUIRES: "pytest"
           # Simple test that requires the project to be build correctly
           CIBW_TEST_COMMAND: >-
-            pytest {project}/tests/test_igzip.py {project}/tests/test_isal.py
+            pytest {project}/tests/test_igzip.py
+            {project}/tests/test_compat.py
+            {project}/tests/test_igzip_lib.py
       - name: Build sdist
         if: "runner.os == 'Linux'"
         run: python setup.py sdist

CHANGELOG.rst

@@ -9,6 +9,9 @@ Changelog
 
 version 0.10.0-dev
 ------------------
++ Added an ``igzip_lib`` module which allows more direct access to ISA-L's
+  igzip_lib API. This allows features such as headerless compression and
+  decompression, as well as setting the memory levels manually.
 + Added more extensive documentation.
 
 version 0.9.0

README.rst

@@ -45,26 +45,34 @@ Acceleration Library (ISA-L) implements several key algorithms in `assembly
 language <https://en.wikipedia.org/wiki/Assembly_language>`_. This includes
 a variety of functions to provide zlib/gzip-compatible compression.
 
-``python-isal`` provides the bindings by offering an ``isal_zlib`` and
-``igzip`` module which are usable as drop-in replacements for the ``zlib``
-and ``gzip`` modules from the stdlib (with some minor exceptions, see below).
+``python-isal`` provides the bindings by offering three modules:
+
++ ``isal_zlib``: A drop-in replacement for the zlib module that uses ISA-L to
+  accelerate its performance.
++ ``igzip``: A drop-in replacement for the gzip module that uses ``isal_zlib``
+  instead of ``zlib`` to perform its compression and checksum tasks, which
+  improves performance.
++ ``igzip_lib``: Provides compression functions which have full access to the
+  API of ISA-L's compression functions.
+
+``isal_zlib`` and ``igzip`` are almost fully compatible with ``zlib`` and
+``gzip`` from the Python standard library. There are some minor differences
+see: differences-with-zlib-and-gzip-modules_.
 
 .. introduction end
 
 Quickstart
 ----------
 
-.. usage start
+.. quickstart start
 
-Python-isal has faster versions of the stdlib's ``zlib`` and ``gzip`` module
-these are called ``isal_zlib`` and ``igzip`` respectively.
-
-They can be imported as follows
+The python-isal modules can be imported as follows
 
 .. code-block:: python
 
     from isal import isal_zlib
     from isal import igzip
+    from isal import igzip_lib
 
 ``isal_zlib`` and ``igzip`` are meant to be used as drop in replacements so
 their api and functions are the same as the stdlib's modules. Except where
@@ -78,7 +86,7 @@ application (just like ``python -m gzip``). Full usage documentation can be
 found on `our readthedocs page <https://python-isal.readthedocs.io>`_.
 
 
-.. usage end
+.. quickstart end
 
 Installation
 ------------
@@ -89,6 +97,8 @@ Installation is supported on Linux, Windows and MacOS. For more advanced
 installation options check the `documentation
 <https://python-isal.readthedocs.io/en/stable/index.html#installation>`_.
 
+.. _differences-with-zlib-and-gzip-modules:
+
 Differences with zlib and gzip modules
 --------------------------------------
 
@@ -144,8 +154,8 @@ This project builds upon the software and experience of many.  Many thanks to:
   to create an extension and helping a novice get start with pointer addresses.
 + The `CPython contributors
   <https://github.com/python/cpython/graphs/contributors>`_.
-  Python-isal mimicks ``zlibmodule.c`` and ``igzip.py`` to make it easier for
-  python users to adopt it.
+  Python-isal mimicks ``zlibmodule.c`` and ``gzip.py`` from the standard
+  library to make it easier for python users to adopt it.
 + `@marcelm <https://github.com/marcelm>`_ for taking a chance on this project
   and make it a dependency for his `xopen
   <https://github.com/pycompression/xopen>`_ and by extension `cutadapt

docs/includes/Installation.rst

@@ -23,28 +23,66 @@ Introduction
 Quickstart
 ==========
 
-Python-isal has faster versions of the stdlib's ``zlib`` and ``gzip`` module
-these are called ``isal_zlib`` and ``igzip`` respectively.
+.. include:: includes/README.rst
+   :start-after: .. quickstart start
+   :end-before: .. quickstart end
+
+============
+Installation
+============
+Installation with pip
+---------------------
+
+::
+
+    pip install isal
+
+Installation is supported on Linux, MacOS and Windows. On most platforms
+wheels are provided.
+The installation will include a staticallly linked version of ISA-L.
+If a wheel is not provided for your system the
+installation will build ISA-L first in a temporary directory. Please check the
+`ISA-L homepage <https://github.com/intel/isa-l>`_ for the build requirements.
+
+The latest development version of python-isal can be installed with::
+
+    pip install git+https://github.com/rhpvorderman/python-isal.git
 
-They can be imported as follows
+This requires having the build requirements installed.
+If you wish to link
+dynamically against a version of libisal installed on your system use::
 
-.. code-block:: python
+     PYTHON_ISAL_LINK_DYNAMIC=true pip install isal --no-binary isal
 
-    from isal import isal_zlib
-    from isal import igzip
+ISA-L is available in numerous Linux distro's as well as on conda via the
+conda-forge channel. Checkout the `ports documentation
+<https://github.com/intel/isa-l/wiki/Ports--Repos>`_ on the ISA-L project wiki
+to find out how to install it. It is important that the development headers
+are also installed.
 
-``isal_zlib`` and ``igzip`` are meant to be used as drop in replacements so
-their api and functions are the same as the stdlib's modules. Except where
-ISA-L does not support the same calls as zlib (See differences below).
+On Debian and Ubuntu the ISA-L libraries (including the development headers)
+can be installed with::
 
-A full API documentation can be found below.
+  sudo apt install libisal-dev
 
-``python -m isal.igzip`` implements a simple gzip-like command line
-application (just like ``python -m gzip``). Full usage documentation can be
-found below.
+Installation via conda
+----------------------
+Python-isal can be installed via conda, for example using
+the `miniconda <https://docs.conda.io/en/latest/miniconda.html>`_ installer
+with a properly setup `conda-forge
+<https://conda-forge.org/docs/user/introduction.html#how-can-i-install-packages-from-conda-forge>`_
+channel. When used with bioinformatics tools setting up `bioconda
+<http://bioconda.github.io/user/install.html#install-conda>`_
+provides a clear set of installation instructions for conda.
 
+python-isal is available on conda-forge and can be installed with::
 
-.. include:: includes/Installation.rst
+  conda install python-isal
+
+This will automatically install the ISA-L library dependency as well, since
+it is available on conda-forge.
+
+.. _differences-with-zlib-and-gzip-modules:
 
 ======================================
 Differences with zlib and gzip modules
@@ -54,11 +92,40 @@ Differences with zlib and gzip modules
    :start-after: .. differences start
    :end-before: .. differences end
 
-.. include:: includes/isal_zlib.rst
+============================
+API Documentation: isal_zlib
+============================
+
+.. automodule:: isal.isal_zlib
+   :members:
+
+========================
+API-documentation: igzip
+========================
+
+.. automodule:: isal.igzip
+   :members: compress, decompress, open
+
+   .. autoclass:: IGzipFile
+      :members:
+      :special-members: __init__
+
+============================
+API Documentation: igzip_lib
+============================
+
+.. automodule:: isal.igzip_lib
+   :members:
+
+==========================
+python -m isal.igzip usage
+==========================
 
-.. include:: includes/igzip.rst
+.. argparse::
+   :module: isal.igzip
+   :func: _argument_parser
+   :prog: python -m isal.igzip
 
-.. include:: includes/isal.igzip_usage.rst
 
 ============
 Contributing

docs/includes/igzip.rst

@@ -41,7 +41,8 @@ class IsalExtension(Extension):
     pass
 
 
-MODULES = [IsalExtension("isal.isal_zlib", ["src/isal/isal_zlib.pyx"])]
+MODULES = [IsalExtension("isal.isal_zlib", ["src/isal/isal_zlib.pyx"]),
+           IsalExtension("isal.igzip_lib", ["src/isal/igzip_lib.pyx"])]
 if SYSTEM_IS_UNIX:
     MODULES.append(IsalExtension("isal._isal", ["src/isal/_isal.pyx"]))
 
@@ -80,7 +81,8 @@ def build_extension(self, ext):
         else:
             if self.compiler.compiler_type == "msvc":
                 compiler = copy.deepcopy(self.compiler)
-                compiler.initialize()
+                if not compiler.initialized:
+                    compiler.initialize()
                 compiler_command = f'"{compiler.cc}"'
                 compiler_args = compiler.compile_options
             elif self.compiler.compiler_type == "unix":

docs/includes/isal.igzip_usage.rst

@@ -31,7 +31,7 @@
 from typing import List, Optional, SupportsInt
 import _compression  # noqa: I201  # Not third-party
 
-from . import isal_zlib
+from . import igzip_lib, isal_zlib
 
 __all__ = ["IGzipFile", "open", "compress", "decompress", "BadGzipFile"]
 
@@ -270,12 +270,11 @@ def compress(data, compresslevel=_COMPRESS_LEVEL_BEST, *, mtime=None):
     Optional argument is the compression level, in range of 0-3.
     """
     header = _create_simple_gzip_header(compresslevel, mtime)
-    # Compress the data without header or trailer in a raw deflate block.
-    compressed = isal_zlib.compress(data, compresslevel, wbits=-15)
-    length = len(data) & 0xFFFFFFFF
-    crc = isal_zlib.crc32(data)
-    trailer = struct.pack("<LL", crc, length)
-    return header + compressed + trailer
+    # use igzip_lib to compress the data without a gzip header but with a
+    # gzip trailer.
+    compressed = igzip_lib.compress(data, compresslevel,
+                                    flag=igzip_lib.COMP_GZIP_NO_HDR)
+    return header + compressed
 
 
 def _gzip_header_end(data: bytes) -> int: