Skip to content

Commit e9757d9

Browse files
: v1: value_mesh basic rle support (meta-pytorch#1466)
Summary: this diff introduces a v0 basic run-length-encoded (RLE) internal representation for `ValueMesh<T>` and updates all APIs to operate transparently over either dense or compressed data. the compression support is purely additive. public semantics are unchanged: a mesh still represents a complete mapping from rank -> value, and iteration, slicing, and region order behave identically (note: complexity of `get()` is O(log k) in compressed mode vs O(1) dense). the compressed form is lossless and idempotent. compression is manual in rust because automatic detection isn't possible in a fully generic type. i originally considered doing it automatically, but rust provides no specialization or reflection to determine whether a given `T` has meaningful equality semantics. for many types (e.g. futures, closures) equality doesn't even exist. because of that, compression must be explicitly invoked via `compress_adjacent_in_place()` or `compress_adjacent_in_place_by(pred)` when the caller knows adjacent elements can be merged. in python, compression happens automatically on construction. for `Py<PyAny>`, equality is defined by pointer identity (`a.as_ptr() == b.as_ptr()`), so adjacent references to the same python object are coalesced into RLE runs. this will produce savings for sentinel-rich, categorical, or boolean data (e.g. repeated `None`, booleans, cpython-interned small integers and strings) but will have little effect for freshly allocated numbers or dynamic objects. Differential Revision: D84169361
1 parent 6242eb9 commit e9757d9

File tree

2 files changed

+412
-25
lines changed

2 files changed

+412
-25
lines changed

0 commit comments

Comments
 (0)