Reduce in Python (#259)

This replaces reduce with a new dictionary UHI syntax. Changes:

* Adds the missing axis=0 shortcut in cpp mode
* Adds a warning to the developer Regular axis shortcut
* Removes the boost_histogram.algorithm module
* Implements a Python interface to indexing using a dict
* Adds a bh.tag.Slicer() shortcut for slicing in the dict

Author: henryiii

CHANGELOG.md

@@ -1,9 +1,15 @@
-### IN DEVELOPMENT
+### Version 0.6
 
 This version fills out most of the remaining features missing from the 0.5.x
-series. The API has changed considerably, providing a more consistent
-experience in Python. The classic API still works in this release, but issues a
-warning and will be removed from the next release.
+series.  You can now use all the storages without the original caveats; even
+the accumulators can be accessed array-at-a-time without copy, pickled quickly,
+and set array-at-a-time, as well.
+
+The API has changed considerably, providing a more consistent experience in
+Python. Most of the classic API still works in this release, but will issue a
+warning and will be removed from the next release. Please use this release to
+transition existing 0.5.x code to the new API.
+
 
 #### User changes
 
@@ -18,7 +24,9 @@ warning and will be removed from the next release.
 * You can now sum over a range with endpoints [#185][]
 * `h.axes` now has the functions from axis as well. [#183][]
 * `bh.project` has become `bh.sum` [#185][]
-* Added `hist.copy()` [#218][]
+* `.reduce(...)` and the reducers in `bh.algorithm` have been removed in favor of dictionary based UHI slicing [#259][]
+* `bh.numpy` module interface updates, `histogram=bh.Histogram` replaces cryptic `bh=True`, and `density=True` is now supported in Numpy mode [#256][]
+* Added `hist.copy()` [#218][] and `hist.shape` [#264][]
 * Signatures are much nicer in Python 3 [#188][]
 * Reprs are better, various properties like `__module__` are now set correctly [#200][]
 
@@ -27,6 +35,7 @@ warning and will be removed from the next release.
 * `.view()` now no longer makes a copy [#194][]
 * Fixes related to string category axis fills [#233][], [#230][]
 * Axes are no longer copies, support setting metadata [#238][], [#246][]
+* Pickling accumulator storages is now comparable in performance simple storages [#258][]
 
 #### Developer changes
 
@@ -58,6 +67,9 @@ warning and will be removed from the next release.
 [#246]: https://github.com/scikit-hep/boost-histogram/pull/246
 [#250]: https://github.com/scikit-hep/boost-histogram/pull/250
 [#255]: https://github.com/scikit-hep/boost-histogram/pull/255
+[#258]: https://github.com/scikit-hep/boost-histogram/pull/258
+[#259]: https://github.com/scikit-hep/boost-histogram/pull/259
+[#264]: https://github.com/scikit-hep/boost-histogram/pull/264
 
 
 ### Version 0.5.2

README.md

@@ -150,7 +150,7 @@ If you are on a Linux system that is not part of the "many" in manylinux, such a
 
 #### Conda-Forge
 
-The boost-histogram package is available on Conda-Forge, as well. All supported versions are available with the exception of Windows + Python 2.7, which cannot build due to the age of the compiler. Please use Pip if you *really* need Python 2.7 on Windows. You will also need the VS 2015 distributable, as described above.
+The boost-histogram package is available on Conda-Forge, as well. All supported versions are available with the exception of Windows + Python 2.7, which cannot built due to the age of the compiler. Please use Pip if you *really* need Python 2.7 on Windows. You will also need the VS 2015 distributable, as described above.
 
 ```
 conda install -c conda-forge boost-histogram
@@ -162,7 +162,7 @@ For a source build, for example from an "sdist" package, the only requirements a
 
 If you are using Python 2.7 on Windows, you will need to use a recent version of Visual studio and force distutils to use it, or just upgrade to Python 3.6 or newer. Check the PyBind11 documentation for [more help](https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2009-builds-on-windows). On some Linux systems, you may need to use a newer compiler than the one your distribution ships with.
 
-Having Numpy before building is recommended (enables multithreaded builds). Boost 1.71 is not required or needed (this only depends on included header-only dependencies).This library is under active development; you can install directly from GitHub if you would like.
+Having Numpy before building is recommended (enables multithreaded builds). Boost is not required or needed (this only depends on included header-only dependencies).This library is under active development; you can install directly from GitHub if you would like.
 
 ```bash
 python -m pip install git+https://github.com/scikit-hep/boost-histogram.git@develop

boost_histogram/__init__.py

@@ -10,7 +10,6 @@
     "axis",
     "storage",
     "accumulators",
-    "algorithm",
     "utils",
     "numpy",
     "loc",
@@ -31,7 +30,7 @@
 
 
 from ._internal.hist import Histogram
-from . import axis, storage, accumulators, algorithm, utils, numpy
+from . import axis, storage, accumulators, utils, numpy
 from .tag import loc, rebin, sum, underflow, overflow
 
 from .version import __version__

boost_histogram/_internal/hist.py

@@ -26,9 +26,12 @@
 
 
 def _arg_shortcut(item):
+    msg = "Developer shortcut: will be removed in a future version"
     if isinstance(item, tuple) and len(item) == 3:
+        warnings.warn(msg, FutureWarning)
         return _core.axis.regular_uoflow(item[0], item[1], item[2], None)
     elif isinstance(item, tuple) and len(item) == 4:
+        warnings.warn(msg, FutureWarning)
         return _core.axis.regular_uoflow(*item)
     elif isinstance(item, Axis):
         return item._ax
@@ -123,7 +126,7 @@ def __init__(self, *axes, **kwargs):
         raise TypeError("Unsupported storage")
 
     def __array__(self):
-        return self.view()
+        return _to_view(self._hist.view(False))
 
     def __add__(self, other):
         return self.__class__(self._hist + other._hist)
@@ -220,6 +223,9 @@ def _axis(self, i):
     def _storage_type(self):
         return cast(self, self._hist._storage_type, Storage)
 
+    def _reduce(self, *args):
+        return self.__class__(self._hist.reduce(*args))
+
 
 # C++ version of histogram
 @set_family(CPP_FAMILY)
@@ -263,9 +269,6 @@ def _empty(self, flow=False):
     def _sum(self, flow=False):
         return self._hist.sum(flow)
 
-    def _reduce(self, *args):
-        return self.__class__(self._hist.reduce(*args))
-
     def _project(self, *args):
         return self.__class__(self._hist.project(*args))
 
@@ -327,18 +330,25 @@ def __repr__(self):
                 ret += " ({0} with flow)".format(outer)
         return ret
 
-    def _compute_commonindex(self, index, expand_ellipsis):
+    def _compute_commonindex(self, index):
+        """
+        Takes indices and returns two iterables; one is a tuple or dict of the
+        original, Ellipsis expanded index, and the other returns index,
+        operation value pairs.
+        """
         # Shorten the computations with direct access to raw object
         hist = self._hist
+
+        # Support dict access
+        if hasattr(index, "items"):
+            return index, index.items()
+
         # Normalize -> h[i] == h[i,]
-        if not isinstance(index, tuple):
+        elif not isinstance(index, tuple):
             index = (index,)
 
         # Now a list
-        if expand_ellipsis:
-            indexes = _expand_ellipsis(index, hist.rank())
-        else:
-            indexes = list(index)
+        indexes = _expand_ellipsis(index, hist.rank())
 
         if len(indexes) != hist.rank():
             raise IndexError("Wrong number of indices for histogram")
@@ -357,7 +367,7 @@ def _compute_commonindex(self, index, expand_ellipsis):
                     raise IndexError("histogram index is out of range")
                 indexes[i] %= hist.axis(i).size
 
-        return indexes
+        return indexes, enumerate(indexes)
 
     def axis(self, i):
         """
@@ -452,21 +462,23 @@ def shape(self):
 
     def __getitem__(self, index):
 
-        indexes = self._compute_commonindex(index, expand_ellipsis=True)
+        indexes, iterator = self._compute_commonindex(index)
 
         # If this is (now) all integers, return the bin contents
-        try:
-            return self._hist.at(*indexes)
-        except RuntimeError:
-            pass
+        # But don't try *dict!
+        if not hasattr(indexes, "items"):
+            try:
+                return self._hist.at(*indexes)
+            except RuntimeError:
+                pass
 
         integrations = set()
         slices = []
         zeroes_start = []
         zeroes_stop = []
 
         # Compute needed slices and projections
-        for i, ind in enumerate(indexes):
+        for i, ind in iterator:
             if not isinstance(ind, slice):
                 raise IndexError(
                     "Invalid arguments as an index, use all integers "
@@ -503,7 +515,7 @@ def __getitem__(self, index):
 
                 slices.append(_core.algorithm.slice_and_rebin(i, begin, end, merge))
 
-        reduced = self.reduce(*slices)
+        reduced = self._reduce(*slices)
         if not integrations:
             return self.__class__(reduced)
         else:
@@ -546,7 +558,7 @@ def __setitem__(self, index, value):
         is 2 larger). Bin edges must be a close match, as well. If you don't
         want this level of type safety, just use ``h[...] = h2.view()``.
         """
-        indexes = self._compute_commonindex(index, expand_ellipsis=True)
+        indexes, iterator = self._compute_commonindex(index)
 
         if isinstance(value, BaseHistogram):
             raise TypeError("Not supported yet")
@@ -566,8 +578,7 @@ def __setitem__(self, index, value):
                 "Setting a histogram with an array must have a matching number of dimensions"
             )
 
-        for n in range(len(indexes)):
-            request = indexes[n]
+        for n, request in iterator:
             has_underflow = self.axes[n].options.underflow
             has_overflow = self.axes[n].options.overflow
 
@@ -605,14 +616,6 @@ def __setitem__(self, index, value):
 
         view[tuple(indexes)] = value
 
-    def reduce(self, *args):
-        """
-        Reduce based on one or more reduce_option's. If you are operating on most
-        or all of your axis, consider slicing with [] notation.
-        """
-
-        return self.__class__(self._hist.reduce(*args))
-
     def project(self, *args):
         """
         Project to a single axis or several axes on a multidiminsional histogram.

boost_histogram/algorithm.py

@@ -16,7 +16,13 @@
     "project",
 )
 
-from ..algorithm import shrink_and_rebin, slice_and_rebin, rebin, shrink, slice
+from .._core.algorithm import shrink_and_rebin, slice_and_rebin, rebin, shrink, slice
+
+shrink_and_rebin.__module__ = "boost_histogram.cpp"
+slice_and_rebin.__module__ = "boost_histogram.cpp"
+rebin.__module__ = "boost_histogram.cpp"
+shrink.__module__ = "boost_histogram.cpp"
+slice.__module__ = "boost_histogram.cpp"
 
 
 def sum(histogram, flow=False):

boost_histogram/cpp/algorithm.py

@@ -2,11 +2,26 @@
 
 del absolute_import, division, print_function
 
-__all__ = ("Locator", "at", "loc", "overflow", "underflow", "rebin", "sum")
+__all__ = ("Slicer", "Locator", "at", "loc", "overflow", "underflow", "rebin", "sum")
 
 import numpy as _np
 
 
+class Slicer(object):
+    """
+    This is a simple class to make slicing inside dictionaries simpler.
+    This is how it should be used:
+
+        s = bh.tag.Slicer()
+
+        h[{0: s[::bh.rebin(2)]}]   # rebin axis 0 by two
+
+    """
+
+    def __getitem__(self, item):
+        return item
+
+
 class Locator(object):
     __slots__ = ("offset",)
 

boost_histogram/tag.py

@@ -3,12 +3,14 @@
 Indexing
 ========
 
-This is the design document for Unified Histogram Indexing (UHI).  Much of the original plan is now implemented in boost-histogram.
-Other histogramming libraries can implement support for this as well, and the "tag" functors, like ``sum`` and ``loc`` can be
-used between libraries.
+This is the design document for Unified Histogram Indexing (UHI).  Much of the
+original plan is now implemented in boost-histogram.  Other histogramming
+libraries can implement support for this as well, and the "tag" functors, like
+``sum`` and ``loc`` can be used between libraries.
 
-The following examples assume you have imported ``loc``, ``sum``, ``rebin``, ``end``, ``underflow``, and ``overflow`` from boost-histogram or any other
-library that implements UHI.
+The following examples assume you have imported ``loc``, ``sum``, ``rebin``,
+``underflow``, and ``overflow`` from boost-histogram or any other library that
+implements UHI.
 
 Access:
 ^^^^^^^
@@ -31,9 +33,9 @@ Slicing:
    h2 = h[loc(v):]       # Slices can be in data coordinates, too
    h2 = h[::rebin(2)]    # Modification operations (rebin)
    h2 = h[a:b:rebin(2)]  # Modifications can combine with slices
-   h2 = h[::sum]     # Projection operations # (name may change)
-   h2 = h[a:b:sum]   # Adding endpoints to projection operations
-   h2 = h[0:end:sum] #   removes under or overflow from the calculation
+   h2 = h[::sum]         # Projection operations # (name may change)
+   h2 = h[a:b:sum]       # Adding endpoints to projection operations
+   h2 = h[0:len:sum]     #   removes under or overflow from the calculation
    h2 = h[a:b, ...]      # Ellipsis work just like normal numpy
 
 Setting
@@ -48,6 +50,7 @@ Setting
 
    h[...] = array(...) # Setting with an array or histogram sets the contents if the sizes match
                        # Overflow can optionally be included if endpoints are left out
+                       # The number of dimensions for non-scalars should match (broadcasting works normally otherwise)
 
 All of this generalizes to multiple dimensions. ``loc(v)`` could return
 categorical bins, but slicing on categories would (currently) not be
@@ -59,9 +62,33 @@ will case the relevant flow bin to be excluded (not currently supported).
 
 ``loc``, ``project``, and ``rebin`` all live inside the histogramming
 package (like boost-histogram), but are completely general and can be created by a
-user using an explicit API (below). ``end``, ``underflow`` and ``overflow`` also
+user using an explicit API (below). ``underflow`` and ``overflow`` also
 follow a general API.
 
+One drawback of the syntax listed above is that it is hard to select an action
+to run on an axis or a few axes out of many. For this use case, you can pass a
+dictionary to the index, and that has the syntax ``{axis:action}``. The actions
+are slices, and follow the rules listed above. This looks like:
+
+.. code:: python
+
+    h[{0: slice(None, None, bh.rebin(2))}] # rebin axis 0 by two
+    h[{1: slice(0, bh.loc(3.5))}]          # slice axis 1 from 0 to the data coordinate 3.5
+    h[{7: slice(0, 2, bh.rebin(4))}]       # slice and rebin axis 7
+
+
+If you don't like manually building slices, you can use the `Slicer()` utility to recover the original slicing syntax inside the dict:
+
+.. code:: python
+
+    s = bh.tag.Slicer()
+
+    h[{0: s[::bh.rebin(2)]}]   # rebin axis 0 by two
+    h[{1: s[0:bh.loc(3.5)]}]   # slice axis 1 from 0 to the data coordinate 3.5
+    h[{7: s[0:2:bh.rebin(4)]}] # slice and rebin axis 7
+
+
+
 Invalid syntax:
 ^^^^^^^^^^^^^^^