DROP op.PARAMS - use funtools.partial() instead...

was letting complex graph designs with constants hard to find
in nested code.

Author: ankostis

README.rst

@@ -41,6 +41,7 @@ produces multiple outputs (``a * b``, ``a - a * b``, and
 ``abs(a - a * b) ** 3``)::
 
    >>> from operator import mul, sub
+   >>> from functools import partial
    >>> from graphtik import compose, operation
 
    >>> # Computes |a|^p.
@@ -53,7 +54,8 @@ Compose the ``mul``, ``sub``, and ``abspow`` functions into a computation graph:
    >>> graphop = compose(name="graphop")(
    ...     operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
    ...     operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
-   ...     operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
+   ...     operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"])
+   ...     (partial(abspow, p=3))
    ... )
 
 

docs/source/composition.rst

@@ -23,6 +23,7 @@ Simple composition of operations
 The simplest use case for ``compose`` is assembling a collection of individual operations into a runnable computation graph.  The example script from :ref:`quick-start` illustrates this well::
 
    >>> from operator import mul, sub
+   >>> from functools import partial
    >>> from graphtik import compose, operation
 
    >>> # Computes |a|^p.
@@ -34,7 +35,8 @@ The simplest use case for ``compose`` is assembling a collection of individual o
    >>> graphop = compose(name="graphop")(
    ...    operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
    ...    operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
-   ...    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
+   ...    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"])
+   ...    (partial(abspow, p=3))
    ... )
 
 The call here to ``compose()`` yields a runnable computation graph that looks like this (where the circles are operations, squares are data, and octagons are parameters):

docs/source/index.rst

@@ -54,6 +54,7 @@ OR with dependencies for plotting support (and you need to install `Graphviz
 Here's a Python script with an example Graphtik computation graph that produces multiple outputs (``a * b``, ``a - a * b``, and ``abs(a - a * b) ** 3``)::
 
    >>> from operator import mul, sub
+   >>> from functools import partial
    >>> from graphtik import compose, operation
 
    # Computes |a|^p.
@@ -66,7 +67,8 @@ Compose the ``mul``, ``sub``, and ``abspow`` functions into a computation graph:
    >>> graphop = compose(name="graphop")(
    ...    operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
    ...    operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
-   ...    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
+   ...    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"])
+   ...    (partial(abspow, p=3))
    ... )
 
 Run the graph-operation and request all of the outputs::

docs/source/operations.rst

@@ -44,6 +44,7 @@ When many operations are composed into a computation graph (see :ref:`graph-comp
 Let's look again at the operations from the script in :ref:`quick-start`, for example::
 
    >>> from operator import mul, sub
+   >>> from functools import partial
    >>> from graphtik import compose, operation
 
    >>> # Computes |a|^p.
@@ -55,25 +56,23 @@ Let's look again at the operations from the script in :ref:`quick-start`, for ex
    >>> graphop = compose(name="graphop")(
    ...    operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
    ...    operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
-   ...    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
+   ...    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"])
+   ...    (partial(abspow, p=3))
    ... )
 
-The ``needs`` and ``provides`` arguments to the operations in this script define a computation graph that looks like this (where the oval are operations, squares/houses are data):
+.. Tip::
+   Notice the use of :func:`functools.partial()` to set parameter ``p`` to a contant value.
+
+The ``needs`` and ``provides`` arguments to the operations in this script define
+a computation graph that looks like this (where the oval are *operations*,
+squares/houses are *data*):
 
 .. image:: images/barebone_3ops.svg
 
 .. Tip::
   See :ref:`plotting` on how to make diagrams like this.
 
 
-Constant operation parameters: ``params``
------------------------------------------
-
-Sometimes an ``operation`` will have a customizable parameter you want to hold constant across all runs of a computation graph.  Usually, this will be a keyword argument of the underlying function.  The ``params`` argument to the ``operation`` constructor provides a mechanism for setting such parameters.
-
-``params`` should be a dictionary whose keys correspond to keyword parameter names from the function underlying an ``operation`` and whose values are passed as constant arguments to those keyword parameters in all computations utilizing the ``operation``.
-
-
 Instantiating operations
 ------------------------
 
@@ -105,31 +104,36 @@ If the functions underlying your computation graph operations are defined elsewh
 
    >>> graphop = compose(name='add_mul_graph')(add_op, mul_op)
 
-The functional specification is also useful if you want to create multiple ``operation`` instances from the same function, perhaps with different parameter values, e.g.::
+The functional specification is also useful if you want to create multiple ``operation``
+instances from the same function, perhaps with different parameter values, e.g.::
 
-   >>> from graphtik import operation, compose
+   >>> from functools import partial
 
    >>> def mypow(a, p=2):
    ...    return a ** p
 
    >>> pow_op1 = operation(name='pow_op1', needs=['a'], provides='a_squared')(mypow)
-   >>> pow_op2 = operation(name='pow_op2', needs=['a'], params={'p': 3}, provides='a_cubed')(mypow)
+   >>> pow_op2 = operation(name='pow_op2', needs=['a'], provides='a_cubed')(partial(mypow, p=3))
 
    >>> graphop = compose(name='two_pows_graph')(pow_op1, pow_op2)
 
 A slightly different approach can be used here to accomplish the same effect by creating an operation "factory"::
 
-   from graphtik import operation, compose
+   >>> def mypow(a, p=2):
+   ...    return a ** p
 
-   def mypow(a, p=2):
-      return a ** p
+   >>> pow_op_factory = operation(mypow, needs=['a'], provides='a_squared')
 
-   pow_op_factory = operation(mypow)
+   >>> pow_op1 = pow_op_factory(name='pow_op1')
+   >>> pow_op2 = pow_op_factory(name='pow_op2', provides='a_cubed', fn=partial(mypow, p=3))
 
-   pow_op1 = pow_op_factory(name='pow_op1', needs=['a'], provides='a_squared')
-   pow_op2 = pow_op_factory(name='pow_op2', needs=['a'], params={'p': 3}, provides='a_cubed')
+   >>> graphop = compose(name='two_pows_graph')(pow_op1, pow_op2)
+   >>> graphop({'a': 2})
+   {'a': 2, 'a_cubed': 8, 'a_squared': 4}
 
-   graphop = compose(name='two_pows_graph')(pow_op1, pow_op2)
+.. Note::
+   You cannot call again the factory to overwrite the *function*,
+   you have to use the ``fn=`` keyword.
 
 
 Modifiers on ``operation`` inputs and outputs

graphtik/base.py

@@ -14,9 +14,7 @@
 log = logging.getLogger(__name__)
 
 
-def jetsam(
-    ex, locs, *salvage_vars: str, annotation="jetsam", **salvage_mappings
-):
+def jetsam(ex, locs, *salvage_vars: str, annotation="jetsam", **salvage_mappings):
     """
     Annotate exception with salvaged values from locals() and raise!
 
@@ -122,7 +120,7 @@ def jetsam(
 class Operation(object):
     """An abstract class representing a data transformation by :meth:`.compute()`."""
 
-    def __init__(self, name=None, needs=None, provides=None, params=None, **kwargs):
+    def __init__(self, name=None, needs=None, provides=None, **kwargs):
         """
         Create a new layer instance.
         Names may be given to this layer and its inputs and outputs. This is
@@ -138,19 +136,12 @@ def __init__(self, name=None, needs=None, provides=None, params=None, **kwargs):
         :param list provides:
             Names of output data objects this provides.
 
-        :param dict params:
-            A dict of key/value pairs representing parameters
-            associated with your operation. These values will be
-            accessible using the ``.params`` attribute of your object.
-            NOTE: It's important that any values stored in this
-            argument must be pickelable.
         """
 
         # (Optional) names for this layer, and the data it needs and provides
         self.name = name
         self.needs = needs
         self.provides = provides
-        self.params = params or {}
 
         # call _after_init as final step of initialization
         self._after_init()
@@ -186,8 +177,8 @@ def compute(self, named_inputs, outputs=None):
     def _after_init(self):
         """
         This method is a hook for you to override. It gets called after this
-        object has been initialized with its ``needs``, ``provides``, ``name``,
-        and ``params`` attributes. People often override this method to implement
+        object has been initialized with its ``name``, ``needs`` and ``provides``
+        attributes. People often override this method to implement
         custom loading logic required for objects that do not pickle easily, and
         for initialization of c++ dependencies.
         """

graphtik/functional.py

@@ -28,13 +28,11 @@ def compute(self, named_inputs, outputs=None):
                 for n in self.needs
                 if isinstance(n, optional) and n in named_inputs
             }
-            # Combine params and optionals into one big glob of keyword arguments.
-            kwargs = {k: v for d in (self.params, optionals) for k, v in d.items()}
 
             # Don't expect sideffect outputs.
             provides = [n for n in self.provides if not isinstance(n, sideffect)]
 
-            results = self.fn(*args, **kwargs)
+            results = self.fn(*args, **optionals)
 
             if not provides:
                 # All outputs were sideffects.
@@ -59,7 +57,7 @@ def compute(self, named_inputs, outputs=None):
                 operation="self",
                 args=lambda locs: {
                     "args": locs.get("args"),
-                    "kwargs": locs.get("kwargs"),
+                    "kwargs": locs.get("optionals"),
                 },
             )
 
@@ -88,10 +86,6 @@ class operation(Operation):
     :param list provides:
         Names of output data objects this operation provides.
 
-    :param dict params:
-        A dict of key/value pairs representing constant parameters
-        associated with your operation.  These can correspond to either
-        ``args`` or ``kwargs`` of ``fn``.
     """
 
     def __init__(self, fn=None, **kwargs):
@@ -119,9 +113,6 @@ def _normalize_kwargs(self, kwargs):
             kwargs["fn"], "__call__"
         ), "operation was not provided with a callable"
 
-        if type(kwargs["params"]) is not dict:
-            kwargs["params"] = {}
-
         return kwargs
 
     def __call__(self, fn=None, **kwargs):
@@ -241,5 +232,5 @@ def __call__(self, *operations, **kwargs):
             net.add_op(op)
 
         return NetworkOperation(
-            net, name=self.name, needs=needs, provides=provides, params={}, **kwargs
+            net, name=self.name, needs=needs, provides=provides, **kwargs
         )

test/test_graphtik.py

@@ -3,6 +3,7 @@
 
 import math
 import sys
+from functools import partial
 from operator import add, floordiv, mul, sub
 from pprint import pprint
 
@@ -56,12 +57,9 @@ def mul_op1(a, b):
 
     # Pow operation
     @operation(
-        name="pow_op1",
-        needs="sum_ab",
-        provides=["sum_ab_p1", "sum_ab_p2", "sum_ab_p3"],
-        params={"exponent": 3},
+        name="pow_op1", needs="sum_ab", provides=["sum_ab_p1", "sum_ab_p2", "sum_ab_p3"]
     )
-    def pow_op1(a, exponent=2):
+    def pow_op1(a, exponent=3):
         return [math.pow(a, y) for y in range(1, exponent + 1)]
 
     assert pow_op1.compute({"sum_ab": 2}, ["sum_ab_p2"]) == {"sum_ab_p2": 4.0}
@@ -117,6 +115,70 @@ def pow_op1(a, exponent=2):
         netop({"sum_ab": 1, "b": 2}, outputs=["b", "bad_node"])
 
 
+def test_network_plan_execute():
+    def powers_in_trange(a, exponent):
+        outputs = []
+        for y in range(1, exponent + 1):
+            p = math.pow(a, y)
+            outputs.append(p)
+        return outputs
+
+    sum_op1 = operation(name="sum1", provides=["sum_ab"], needs=["a", "b"])(add)
+    mul_op1 = operation(name="mul", provides=["sum_ab_times_b"], needs=["sum_ab", "b"])(
+        mul
+    )
+    pow_op1 = operation(
+        name="pow",
+        needs=["sum_ab", "exponent"],
+        provides=["sum_ab_p1", "sum_ab_p2", "sum_ab_p3"],
+    )(powers_in_trange)
+    sum_op2 = operation(
+        name="sum2", provides=["p1_plus_p2"], needs=["sum_ab_p1", "sum_ab_p2"]
+    )(add)
+
+    net = network.Network()
+    net.add_op(sum_op1)
+    net.add_op(mul_op1)
+    net.add_op(pow_op1)
+    net.add_op(sum_op2)
+    net.compile()
+
+    #
+    # Running the network
+    #
+
+    # get all outputs
+    exp = {
+        "a": 1,
+        "b": 2,
+        "exponent": 3,
+        "p1_plus_p2": 12.0,
+        "sum_ab": 3,
+        "sum_ab_p1": 3.0,
+        "sum_ab_p2": 9.0,
+        "sum_ab_p3": 27.0,
+        "sum_ab_times_b": 6,
+    }
+
+    inputs = {"a": 1, "b": 2, "exponent": 3}
+    plan = net.compile(outputs=None, inputs=inputs.keys())
+    sol = plan.execute(named_inputs=inputs)
+    assert sol == exp
+
+    # get specific outputs
+    exp = {"sum_ab_times_b": 6}
+    plan = net.compile(outputs=["sum_ab_times_b"], inputs=list(inputs))
+    sol = plan.execute(named_inputs=inputs)
+    assert sol == exp
+
+    # start with inputs already computed
+    inputs = {"sum_ab": 1, "b": 2, "exponent": 3}
+    exp = {"sum_ab_times_b": 2}
+    plan = net.compile(outputs=["sum_ab_times_b"], inputs=inputs)
+    sol = plan.execute(named_inputs={"sum_ab": 1, "b": 2})
+    assert sol == exp
+
+
 def test_network_simple_merge():
 
     sum_op1 = operation(name="sum_op1", needs=["a", "b"], provides="sum1")(add)
@@ -178,11 +240,8 @@ def test_network_merge_in_doctests():
         operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
         operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
         operation(
-            name="abspow1",
-            needs=["a_minus_ab"],
-            provides=["abs_a_minus_ab_cubed"],
-            params={"p": 3},
-        )(abspow),
+            name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"]
+        )(partial(abspow, p=3)),
     )
 
     another_graph = compose(name="another_graph")(
@@ -853,11 +912,8 @@ def test_multithreading_plan_execution():
         operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
         operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
         operation(
-            name="abspow1",
-            needs=["a_minus_ab"],
-            provides=["abs_a_minus_ab_cubed"],
-            params={"p": 3},
-        )(abspow),
+            name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"]
+        )(partial(abspow, p=3)),
     )
 
     pool = Pool(10)
@@ -969,11 +1025,8 @@ def test_compose_another_network(bools):
         operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
         operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
         operation(
-            name="abspow1",
-            needs=["a_minus_ab"],
-            provides=["abs_a_minus_ab_cubed"],
-            params={"p": 3},
-        )(abspow),
+            name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"]
+        )(partial(abspow, p=3)),
     )
     if parallel1:
         graphop.set_execution_method("parallel")