-
Notifications
You must be signed in to change notification settings - Fork 78
Commit d403dcf
: v1: value_mesh basic rle support (#1466)
Summary:
this diff introduces a v0 basic run-length-encoded (RLE) internal representation for `ValueMesh<T>` and updates all APIs to operate transparently over either dense or compressed data. the compression support is purely additive. public semantics are unchanged: a mesh still represents a complete mapping from rank -> value, and iteration, slicing, and region order behave identically (note: complexity of `get()` is O(log k) in compressed mode vs O(1) dense). the compressed form is lossless and idempotent.
compression is manual in rust because automatic detection isn't possible in a fully generic type. i originally considered doing it automatically, but rust provides no specialization or reflection to determine whether a given `T` has meaningful equality semantics. for many types (e.g. futures, closures) equality doesn't even exist. because of that, compression must be explicitly invoked via `compress_adjacent_in_place()` or `compress_adjacent_in_place_by(pred)` when the caller knows adjacent elements can be merged.
in python, compression happens automatically on construction. for `Py<PyAny>`, equality is defined by pointer identity (`a.as_ptr() == b.as_ptr()`), so adjacent references to the same python object are coalesced into RLE runs. this will produce savings for sentinel-rich, categorical, or boolean data (e.g. repeated `None`, booleans, cpython-interned small integers and strings) but will have little effect for freshly allocated numbers or dynamic objects.
Differential Revision: D841693611 parent 6c266bf commit d403dcfCopy full SHA for d403dcf
File tree
Expand file treeCollapse file tree
2 files changed
+412
-25
lines changedFilter options
- hyperactor_mesh/src/v1
- monarch_hyperactor/src
Expand file treeCollapse file tree
2 files changed
+412
-25
lines changed
0 commit comments