spec: add Matrix datatype (#8)

Author: birkenfeld

protocol/issues/076 Interface for Measurable hardware.rst

@@ -34,35 +34,35 @@ Proposal
 It is proposed to add three new interface classes.
 
 
-``MeasurableController`` (no base interface)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``Controller`` (no base interface)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Accessibles:
 
 ``status``
     Mandatory: Standard SECoP status.
     The module is in ``BUSY`` state while the measurable is acquiring.
-    The module is in ``PREPARED`` state after ``prepare()`` is called or data
-    acquisition is paused by ``hold()``.
+    The module is in ``PREPARED`` state after ``prepare`` is called or data
+    acquisition is paused by ``hold``.
 
-``prepare()``
+command ``prepare``
     Optional command: prepares a data acquisition so that triggering with ``go``
     is immediate.  No-op if already prepared.  Cannot be called when busy.
 
-``go()``
+command ``go``
     Mandatory command: starts a data acquisition.  No-op if busy.
     Data acquisition runs until one of the channels' active presets is hit or
-    ``stop`` is called explicitly.  Runs the ``prepare()`` sequence first if
+    ``stop`` is called explicitly.  Runs the ``prepare`` sequence first if
     module is not already prepared.
 
-``hold()``
+command ``hold``
     Optional command: pauses a data acquisition.  No-op if not busy.
-    Subsequent ``go()`` continues the acquisition without clearing currently
+    Subsequent ``go`` continues the acquisition without clearing currently
     acquired data.
 
-``stop()``
+command ``stop``
     Optional command: stops a data acquisition.  No-op if not busy.
-    Subsequent ``go()`` starts a new acquisition with clearing currently
+    Subsequent ``go`` starts a new acquisition with clearing currently
     acquired data.
 
 
@@ -116,10 +116,10 @@ Accessibles:
 ``roi``
     Optional: a list containing a ``(min, max)`` tuple per dimension, to specify
     a sub-slice of matrix data to consider in ``value`` and return in
-    ``get_data()``.
+    ``get_data``.
 
-``get_data()``
-    Optional: returns the channel's matrix data, see below for details.
+command ``get_data``
+    Optional: returns the channel's data, with a ``matrix`` data type.
 
 The ``value`` parameter only contains a useful "reduced" form of the data, for
 example, the sum of all events in the matrix, or the average of all intensity
@@ -135,38 +135,6 @@ data is considered for this reduction.
     useful (i.e. not just "pixel 1..N").  (Precise semantics to be specified.)
 
 
-The return value of ``get_data()``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Since the returned data matrix can get pretty large, it is not efficient to
-encode it as nested ``Array``\s.  Instead, the data is kept in Blob form
-(base64-encoded for JSON transmission).
-
-The return value is a struct, where the following members are currently
-specified:
-
-- ``data``: The data as a ``Blob``, whose datainfo must have these additional
-  properties:
-
-  - ``elementtype``: The type (and size, and byte order) of each matrix element,
-    a string in Numpy convention (e.g. ``"<u4"``).
-  - ``names``: A list of names for each dimension
-  - ``maxlengths``: A list of maximum lengths for each dimension
-
-- ``dims``: An array containing the actual lengths of each dimension.
-
-The order of the matrix elements is defined so that the dimension with the
-fastest running index comes first in ``dims``, ``names`` and ``maxlengths``.
-
-Example: ``data`` is ``[1, 2, 3, 4, 5, 6]``, ``dims`` is ``[2, 3]`` and
-``names`` is ``["x", "y"]``.  Then the matrix looks as follows::
-
-  .     x=0 x=1
-  y=0   1   2
-  y=1   3   4
-  y=2   5   6
-
-
 Remarks
 ~~~~~~~
 

protocol/specification/datainfo.rst

@@ -13,7 +13,8 @@ SECoP defines some basic data types for numeric quantities, like double_,
 scaled_ and int_.  An enum_ is defined for convenience of not having to remember
 the meaning of values from a reduced set.  A bool_ datatype is similar to a
 predefined Enum, but uses the JSON values true and false.  For non-numeric
-types, a string_ and a blob_ are defined as well.
+types, a string_ and a blob_ are defined as well.  Finally, matrix_ can
+transport larger multi-dimensional homogeneous arrays.
 
 Furthermore, SECoP not only defines basic data types, but also structured
 datatypes.  tuple_ allows aggregation of a fixed amount of values with different
@@ -468,6 +469,75 @@ Example: ``{"x": 0.5, "y": 1}``
 Related issue: :issue:`035 Partial Structs`
 
 
+.. _matrix:
+
+Binary Matrix: ``matrix``
+-------------------------
+
+Type for transferring a medium to large amount of homogeneous arrays with
+potentially multiple dimensions.
+
+At the moment, the type intends direct transfer of the data within the JSON
+data.  It could be extended later to allow referring to a side-channel for
+obtaining the data.
+
+Mandatory Data Properties
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``"names"``
+    A list of names for each dimension in the data.
+
+``"maxlen"``
+    A list of maximum lengths for each dimension. The actual lengths can vary
+    but may not exceed these limits.
+
+``"elementtype"``
+    A string defining the type of each element, as a combination of three parts:
+
+    - ``<`` or ``>`` to indicate little or big endianness.
+    - ``i``, ``u``, ``f`` to indicate signed or unsigned integers or floating
+      point numbers.
+    - a number to indicate the number of bytes per element (1, 2, 4 or 8).
+
+    Example: ``"<u4"`` is a little-endian encoded 32-bit unsigned integer.
+
+``"compression"``
+    A string defining if and how the data is each ``blob`` is compressed.
+    Currently, no compression types are defined.
+
+
+Example
+~~~~~~~
+``{"type": "matrix", "elementtype": "<f4", "names": ["x", "y"], "maxlen": [100, 100]}``
+
+
+Transport
+~~~~~~~~~
+
+As a JSON object containing the following items:
+
+``"len"``
+    List of the actual length of each dimension in the data.
+
+``"blob"``
+    The data, encoded as a single-line base64 (see :RFC:`4648`) encoded
+    JSON-string.
+
+Example: ``{"len": [2, 3], "blob": "AACAPwAAAEAAAEBAAACAQAAAoEAAAMBA"}``
+
+The order of the matrix elements is defined so that the first dimension
+named in ``names`` (and listed in ``maxlen``/``len``) varies the fastest.
+
+In this example, the result of decoding ``blob`` as a flat sequence of 4-byte
+floats is ``[1, 2, 3, 4, 5, 6]``.  Then the matrix looks as follows::
+
+  .     x=0 x=1
+  y=0   1   2
+  y=1   3   4
+  y=2   5   6
+
+
+
 .. _command:
 
 Commands: ``command``

protocol/specification/overview.rst

@@ -130,6 +130,7 @@ example:
 - :ref:`Enum <enum>`
 - :ref:`String <string>`
 - :ref:`Blob <blob>`
+- :ref:`Matrix <matrix>`
 
 For more complicated values, there are three structured datatypes: