Add optional reference (production) exchange column to datapackage schema (#108)

Adds an optional per-exchange `reference` boolean column, stored as its own
side-resource (`kind="reference"`) exactly like `flip`/`rescale`: additive,
backward compatible, and written only when at least one entry is flagged.

Modellers can now record which technosphere exchange is the reference
(production) exchange, so consumers such as bw_graph_tools can read it
directly instead of relying on structural heuristics, which cannot
disambiguate co-production columns (an activity with multiple same-sign
outputs whose products also appear in other columns).

- MatrixEntry.reference / ArrayEntry.reference
- reference_array kwarg on all add_* methods + _add_reference_array_resource
- threaded through dictionary_formatter / resolve_dict_iterator
- "reference" added to Parquet kind mapping and merging suffix whitelist
- README section, CHANGES entry, version bump 1.5 -> 1.6
- tests: dataclass defaults/validation, low-level validation, skip-when-empty,
  write->reload round-trip

Refs cauldron/brightway-api#739

Author: cmutel

CHANGES.md

@@ -1,5 +1,9 @@
 # `bw_processing` Changelog
 
+## [1.6] - 2026-07-10
+
+* Add optional `reference` boolean column marking reference (production) exchanges. Stored as a `reference_array` side-resource (`kind="reference"`) analogous to `flip`, and exposed on `MatrixEntry`/`ArrayEntry` and all `add_*` methods. Lets modellers record the reference exchange explicitly instead of relying on `bw_graph_tools`' structural heuristics, which cannot disambiguate co-production columns. See cauldron/brightway-api#739.
+
 ## [1.5] - 2026-06-04
 
 * [PR #100: Deduplicate chunked bucket-fill logic; fix #95 and #97](https://github.com/brightway-lca/bw_processing/pull/100)

README.md

@@ -168,12 +168,12 @@ print(data_obj.url)
 
 ### Scale arrays
 
-Any resource group (persistent or dynamic, vector or array) can carry an optional `scale_array`: a one-dimensional float array of the same length as `indices_array`. Each element is a multiplicative factor applied to the corresponding data value before it is inserted into the matrix. The factor is applied to both static and stochastically-sampled values. A value of `1.0` leaves the data unchanged.
+Any resource group (persistent or dynamic, vector or array) can carry an optional `rescale_array`: a one-dimensional float array of the same length as `indices_array`. Each element is a multiplicative factor applied to the corresponding data value before it is inserted into the matrix. The factor is applied to both static and stochastically-sampled values. A value of `1.0` leaves the data unchanged.
 
 Typical use cases:
 
-* **Allocation factors** — when a process produces multiple products, the exchange amounts must be partitioned between them. Storing the allocation coefficients as a `scale_array` keeps them alongside the data they modify without requiring a separate processing step.
-* **Unit conversions** — when source data is expressed in a unit that differs from the matrix convention, a constant conversion factor can be stored as a `scale_array` rather than baked into every data value.
+* **Allocation factors** — when a process produces multiple products, the exchange amounts must be partitioned between them. Storing the allocation coefficients as a `rescale_array` keeps them alongside the data they modify without requiring a separate processing step.
+* **Unit conversions** — when source data is expressed in a unit that differs from the matrix convention, a constant conversion factor can be stored as a `rescale_array` rather than baked into every data value.
 
 ```python
 import numpy as np
@@ -183,18 +183,45 @@ from bw_processing.constants import INDICES_DTYPE
 dp = create_datapackage()
 indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
 data_array = np.array([100.0, 200.0, 300.0])
-scale_array = np.array([0.6, 1.0, 0.4])  # e.g. allocation factors
+rescale_array = np.array([0.6, 1.0, 0.4])  # e.g. allocation factors
 
 dp.add_persistent_vector(
     matrix="technosphere",
     name="my-process",
     indices_array=indices_array,
     data_array=data_array,
-    scale_array=scale_array,
+    rescale_array=rescale_array,
 )
 ```
 
-The stored resource has `kind="scale"` and can be retrieved via `dp.get_resource("my-process.scale")`. The `scale_array` must be a float dtype (`float32` or `float64`); passing an integer array raises `WrongDatatype`.
+The stored resource has `kind="rescale"` and can be retrieved via `dp.get_resource("my-process.rescale")`. The `rescale_array` must be a float dtype (`float32` or `float64`); passing an integer array raises `WrongDatatype`.
+
+### Reference (production) exchanges
+
+Any resource group can also carry an optional `reference_array`: a one-dimensional boolean array of the same length as `indices_array`. Where an element is `True`, that exchange is the **reference (production) exchange** for its activity/column.
+
+The five structural heuristics in `bw_graph_tools` (matching ids, single non-flipped entry, single positive, single negative, unique product) cannot always identify the reference exchange — whenever an activity has more than one same-sign exchange and the products also appear in other columns, the choice is genuinely ambiguous. Only the modeller knows the answer. `reference_array` records it directly so downstream tools can read it instead of guessing.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage
+from bw_processing.constants import INDICES_DTYPE
+
+dp = create_datapackage()
+indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+data_array = np.array([1.0, 0.5, 2.0])
+reference_array = np.array([True, False, False])  # first exchange is the reference
+
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=indices_array,
+    data_array=data_array,
+    reference_array=reference_array,
+)
+```
+
+The stored resource has `kind="reference"` and can be retrieved via `dp.get_resource("my-process.reference")`. It must be a boolean array; passing a non-boolean array raises `WrongDatatype`. To keep the common case cheap, the resource is written only when at least one entry is `True` — a group with no reference flags carries no `reference` resource. Using the high-level `MatrixEntry`/`ArrayEntry` API, set `reference=True` (or a boolean `reference` array) on the entries you want flagged.
 
 ### Parameter arrays for sensitivity analysis
 

src/bw_processing/__init__.py

@@ -35,7 +35,7 @@
     "UndefinedInterface",
 )
 
-__version__ = "1.5"
+__version__ = "1.6"
 
 
 from bw_processing.array_creation import create_array, create_structured_array

src/bw_processing/datapackage.py

@@ -512,6 +512,7 @@ def add_persistent_vector_from_iterator(
             distributions_array,
             flip_array,
             rescale_array,
+            reference_array,
         ) = resolve_dict_iterator(dict_iterator, nrows)
         self.add_persistent_vector(
             matrix=matrix,
@@ -522,6 +523,7 @@ def add_persistent_vector_from_iterator(
             flip_array=flip_array,
             distributions_array=distributions_array,
             rescale_array=rescale_array,
+            reference_array=reference_array,
             matrix_serialize_format_type=matrix_serialize_format_type,
             **kwargs,
         )
@@ -538,6 +540,8 @@ def add_entries(
         High-level convenience method that does not require working directly
         with NumPy arrays. If any entry has a ``rescale`` value other than
         ``1.0``, the rescale values are stored as a ``rescale_array`` resource.
+        If any entry has ``reference=True``, the reference flags are stored as
+        a ``reference_array`` resource (``kind="reference"``).
 
         Args:
             matrix: Name of the target matrix (e.g. ``"technosphere"``).
@@ -562,6 +566,8 @@ def add_array_entries(
         Each :class:`.ArrayEntry` becomes one persistent-array resource group.
         Resource group names are auto-generated. If an entry has a ``rescale``
         array it is stored as a ``rescale_array`` resource (``kind="rescale"``).
+        If an entry has a ``reference`` array with any ``True`` value it is
+        stored as a ``reference_array`` resource (``kind="reference"``).
 
         Args:
             matrix: Name of the target matrix (e.g. ``"technosphere"``).
@@ -577,6 +583,7 @@ def add_array_entries(
                 data_array=entry.data,
                 flip_array=entry.flip,
                 rescale_array=entry.rescale,
+                reference_array=entry.reference,
             )
 
     def add_persistent_vector(
@@ -589,6 +596,7 @@ def add_persistent_vector(
         flip_array: Optional[np.ndarray] = None,
         distributions_array: Optional[np.ndarray] = None,
         rescale_array: Optional[np.ndarray] = None,
+        reference_array: Optional[np.ndarray] = None,
         params_array: Optional[np.ndarray] = None,
         param_labels: Optional[list] = None,
         param_label_schema: Optional[AnyLabelSchema] = None,
@@ -605,6 +613,12 @@ def add_persistent_vector(
         factors and unit conversions.  A value of ``1.0`` leaves the data
         unchanged.
 
+        ``reference_array`` is an optional 1-D boolean array of the same length
+        as ``indices_array``.  Where ``True``, that entry is the reference
+        (production) exchange for its column.  It is stored as a
+        ``reference_array`` resource (``kind="reference"``) only when at least
+        one entry is flagged.
+
         ``params_array`` is an optional 1-D float array recording the values of
         independent variables (e.g. model parameters) used to generate this
         resource group.  ``param_labels`` is an optional list of label objects
@@ -713,6 +727,15 @@ def add_persistent_vector(
                 matrix_serialize_format_type=matrix_serialize_format_type,
                 **kwargs,
             )
+        if reference_array is not None:
+            self._add_reference_array_resource(
+                reference_array=reference_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
         if params_array is not None:
             params_array = load_bytes(params_array)
             if params_array.ndim != 1:
@@ -751,6 +774,7 @@ def add_persistent_array(
         name: Optional[str] = None,
         flip_array: Optional[np.ndarray] = None,
         rescale_array: Optional[np.ndarray] = None,
+        reference_array: Optional[np.ndarray] = None,
         params_array: Optional[np.ndarray] = None,
         param_labels: Optional[list] = None,
         param_label_schema: Optional[AnyLabelSchema] = None,
@@ -767,6 +791,12 @@ def add_persistent_array(
         factors and unit conversions.  A value of ``1.0`` leaves the data
         unchanged.
 
+        ``reference_array`` is an optional 1-D boolean array of the same length
+        as ``indices_array``.  Where ``True``, that entry is the reference
+        (production) exchange for its column.  It is stored as a
+        ``reference_array`` resource (``kind="reference"``) only when at least
+        one entry is flagged.
+
         ``params_array`` is an optional 2-D float array of shape
         ``(n_params, n_scenarios)`` where ``n_scenarios`` must equal
         ``data_array.shape[1]``.  It records the independent variable values
@@ -850,6 +880,15 @@ def add_persistent_array(
                 matrix_serialize_format_type=matrix_serialize_format_type,
                 **kwargs,
             )
+        if reference_array is not None:
+            self._add_reference_array_resource(
+                reference_array=reference_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
         if params_array is not None:
             params_array = load_bytes(params_array)
             if params_array.ndim != 2:
@@ -912,7 +951,7 @@ def write_modified(self):
                     if kind == "indices":
                         meta_object = "vector"
                         meta_type = "indices"
-                    elif kind in ("flip", "rescale", "params"):
+                    elif kind in ("flip", "rescale", "reference", "params"):
                         meta_object = "vector"
                         meta_type = "generic"
                     elif kind == "distributions":
@@ -986,6 +1025,41 @@ def _add_rescale_array_resource(
             **kwargs,
         )
 
+    def _add_reference_array_resource(
+        self,
+        *,
+        reference_array: np.ndarray,
+        indices_array: np.ndarray,
+        name: str,
+        keep_proxy: bool,
+        matrix_serialize_format_type: Optional[MatrixSerializeFormat],
+        **kwargs,
+    ) -> None:
+        reference_array = load_bytes(reference_array)
+        if reference_array.dtype != bool:
+            raise WrongDatatype(
+                "`reference_array` dtype is {}, but must be `bool`".format(reference_array.dtype)
+            )
+        if reference_array.shape != indices_array.shape:
+            raise ShapeMismatch(
+                "`reference_array` shape ({}) doesn't match `indices_array` ({}).".format(
+                    reference_array.shape, indices_array.shape
+                )
+            )
+        # If no references flagged, don't need to store it
+        if reference_array.sum():
+            self._add_numpy_array_resource(
+                array=reference_array,
+                group=name,
+                name=name + ".reference",
+                kind="reference",
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                meta_object="vector",
+                meta_type="generic",
+                **kwargs,
+            )
+
     @staticmethod
     def _check_params_args(
         params_array: Optional[np.ndarray],
@@ -1133,6 +1207,7 @@ def add_dynamic_vector(
         name: Optional[str] = None,
         flip_array: Optional[np.ndarray] = None,  # Not interface
         rescale_array: Optional[np.ndarray] = None,  # Not interface
+        reference_array: Optional[np.ndarray] = None,  # Not interface
         params_array: Optional[np.ndarray] = None,  # Not interface
         param_labels: Optional[list] = None,
         param_label_schema: Optional[AnyLabelSchema] = None,
@@ -1147,7 +1222,7 @@ def add_dynamic_vector(
         1-D numpy array of length ``len(indices_array)`` each time it is called.
 
         The ``indices_array``, optional ``flip_array``, optional ``rescale_array``,
-        and optional ``params_array`` are static and are stored as normal numpy
+        optional ``reference_array``, and optional ``params_array`` are static and are stored as normal numpy
         resources.  See ``add_persistent_vector`` for documentation of the
         ``params_array``, ``param_labels``, and ``param_label_schema`` arguments.
 
@@ -1162,6 +1237,8 @@ def add_dynamic_vector(
                 multiplied by ``-1`` before insertion.
             rescale_array: Optional 1-D float array of multiplicative factors
                 applied before matrix insertion.
+            reference_array: Optional 1-D boolean array; where ``True`` the
+                entry is the reference (production) exchange for its column.
             keep_proxy: If ``True``, store a proxy rather than the raw array
                 for on-disk resources.
             matrix_serialize_format_type: Override the instance-level
@@ -1219,6 +1296,15 @@ def add_dynamic_vector(
                 matrix_serialize_format_type=matrix_serialize_format_type,
                 **kwargs,
             )
+        if reference_array is not None:
+            self._add_reference_array_resource(
+                reference_array=reference_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
         if params_array is not None:
             params_array = load_bytes(params_array)
             if params_array.ndim != 1:
@@ -1267,6 +1353,7 @@ def add_dynamic_array(
         name: Optional[str] = None,
         flip_array: Optional[np.ndarray] = None,
         rescale_array: Optional[np.ndarray] = None,  # Not interface
+        reference_array: Optional[np.ndarray] = None,  # Not interface
         params_array: Optional[np.ndarray] = None,  # Not interface
         param_labels: Optional[list] = None,
         param_label_schema: Optional[AnyLabelSchema] = None,
@@ -1283,7 +1370,7 @@ def add_dynamic_array(
         interface.
 
         The ``indices_array``, optional ``flip_array``, optional ``rescale_array``,
-        and optional ``params_array`` are static and are stored as normal numpy
+        optional ``reference_array``, and optional ``params_array`` are static and are stored as normal numpy
         resources.  For dynamic arrays the column count of ``params_array`` is
         not validated against the interface (whose column count may be unknown at
         write time).  See ``add_persistent_vector`` for documentation of the
@@ -1300,6 +1387,8 @@ def add_dynamic_array(
                 multiplied by ``-1`` before insertion.
             rescale_array: Optional 1-D float array of multiplicative factors
                 applied before matrix insertion.
+            reference_array: Optional 1-D boolean array; where ``True`` the
+                entry is the reference (production) exchange for its column.
             keep_proxy: If ``True``, store a proxy rather than the raw array
                 for on-disk resources.
             matrix_serialize_format_type: Override the instance-level
@@ -1365,6 +1454,15 @@ def add_dynamic_array(
                 matrix_serialize_format_type=matrix_serialize_format_type,
                 **kwargs,
             )
+        if reference_array is not None:
+            self._add_reference_array_resource(
+                reference_array=reference_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
         if params_array is not None:
             params_array = load_bytes(params_array)
             if params_array.ndim != 2:

src/bw_processing/matrix_entry.py

@@ -63,6 +63,12 @@ class MatrixEntry:
             Stored as a ``rescale_array`` resource (``kind="rescale"``). Note
             that the Python ``float`` value is downcast to ``numpy.float32``
             when written to the structured array.
+        reference: If True, this exchange is the reference (production) exchange
+            for its activity/column. Consumers such as bw_graph_tools use this
+            to identify production exchanges directly instead of guessing from
+            matrix structure. Stored as a ``reference_array`` resource
+            (``kind="reference"``) only when at least one entry is flagged;
+            defaults to False.
     """
 
     row: int
@@ -77,6 +83,7 @@ class MatrixEntry:
     maximum: float = math.nan
     negative: bool = False
     rescale: float = 1.0
+    reference: bool = False
 
     def __post_init__(self):
         if self.uncertainty_type in _NO_UNCERTAINTY_IDS:
@@ -109,13 +116,18 @@ class ArrayEntry:
         rescale: Optional 1-D float array of per-entry multiplicative factors
             (one per row). ``1.0`` leaves the value unchanged. Stored as a
             ``rescale_array`` resource (``kind="rescale"``).
+        reference: Optional 1-D boolean sequence of length ``n_entries``.
+            Where True, that entry is the reference (production) exchange for
+            its column. Stored as a ``reference_array`` resource
+            (``kind="reference"``) only when at least one entry is flagged.
     """
 
     rows: np.ndarray
     cols: np.ndarray
     data: np.ndarray
     flip: Optional[np.ndarray] = None
     rescale: Optional[np.ndarray] = None
+    reference: Optional[np.ndarray] = None
 
     def __post_init__(self):
         self.rows = np.asarray(self.rows)
@@ -150,6 +162,12 @@ def __post_init__(self):
                 raise ValueError(
                     f"`rescale` shape {self.rescale.shape} doesn't match `rows` shape {self.rows.shape}"
                 )
+        if self.reference is not None:
+            self.reference = np.asarray(self.reference, dtype=bool)
+            if self.reference.shape != self.rows.shape:
+                raise ValueError(
+                    f"`reference` shape {self.reference.shape} doesn't match `rows` shape {self.rows.shape}"
+                )
 
 
 def create_datapackage_from_entries(

src/bw_processing/merging.py

@@ -53,7 +53,7 @@ def add_resource_suffix(metadata: dict, suffix: str) -> dict:
     last = metadata["name"].split(".")[-1]
     rest = metadata["name"][: -len(last) - 1]
 
-    if last not in {"indices", "data", "distributions", "flip"}:
+    if last not in {"indices", "data", "distributions", "flip", "rescale", "reference"}:
         raise ValueError("Can't understand resource name suffix")
 
     rest = metadata["name"][: -len(last) - 1]