Merge pull request #64 from triblespace/codex/evaluate-features-for-bytesarena

docs: add ByteArea metadata examples

Author: somethingelseentirely

CHANGELOG.md

@@ -1,9 +1,54 @@
 # Changelog
 
 ## Unreleased
-- Added `WaveletMatrix::to_bytes` and `WaveletMatrix::from_bytes` returning metadata and bytes for zero-copy persistence.
-- Documented the serialized `WaveletMatrix` layout with ASCII art.
-- Added `CompactVector::to_bytes` and `from_bytes` for zero-copy serialization.
+- Introduced a `Serializable` trait for metadata-based reconstruction and
+  implemented it for `CompactVector`, `DacsByte`, and `WaveletMatrix`.
+- Audited `DacsByte` and `WaveletMatrix` to leverage `SectionHandle::view`
+  during deserialization, removing legacy `slice_to_bytes` helpers and fully
+  adopting the `ByteArea`-backed reconstruction path.
+- Switched internal bit-vector words and handles from `usize` to `u64`, removing
+  unsafe handle transmutes in `WaveletMatrixBuilder` and fixing word size to
+  64-bit.
+  - Reversed remaining layers and popped in `WaveletMatrixBuilder::freeze`
+    to avoid repeated vector shifts.
+  - `WaveletMatrixMeta` now stores a handle slice of per-layer handles, and
+    `WaveletMatrixBuilder` allocates that slice from the `SectionWriter`.
+  - `WaveletMatrixBuilder::with_capacity` records each layer's handle up front,
+    eliminating handle assignment during `freeze`.
+  - Switched to the zerocopy `SectionHandle` from `anybytes`, removing the
+    interim `HandleRepr` shim.
+  - Added `WaveletMatrixBuilder` for fixed-size construction, writing raw bits per
+    layer and stably partitioning them on `freeze`; `WaveletMatrix::from_iter`
+    now builds via this builder without requiring iterator cloning.
+  - `WaveletMatrix` construction now goes through `from_iter`, which allocates
+    layer bitvectors from a `SectionWriter` and consumes a single
+    `ExactSizeIterator` without temporary `CompactVector` partitions.
+  - `CompactVector::iter` now implements `ExactSizeIterator` to support the new
+    constructor.
+  - `WaveletMatrixBuilder::freeze` partitions layers in place, removing the
+    temporary bit buffer previously used during construction.
+  - Removed `order` and `next_order` buffers by sorting remaining layers in
+    place during each `freeze` step.
+  - Optimized `WaveletMatrixBuilder::freeze` using stable per-layer partitions
+    and cycle-based permutations, reducing layer processing to linear time.
+  - Replaced the `perm` array with a scratch `visited` bitmap and cycle
+    rotations so each level permutes lower layers in place with only `O(n)`
+    extra bits.
+  - Stored row suffix bits in a `usize` during cycle rotations, removing the
+    temporary `Vec<bool>` from `rotate_cycle_over_lower_levels`.
+  - Reused `BitVectorBuilder` as the scratch `visited` bitmap for
+    wavelet-matrix construction, eliminating the separate `BitArrayBuilder`.
+  - Added `swap_bits` helper to `BitVectorBuilder` for in-place bit exchanges.
+  - Reworked `WaveletMatrix::from_iter` to require a cloneable iterator and
+    build layers in two passes without temporary buffers.
+- Rewrote `CompactVectorBuilder` to use fixed-size `set_int` and `set_ints`
+  APIs, removing `push_int`/`extend` and updating builders and examples.
+- Added `with_capacity` constructor on `BitVectorBuilder` and honored capacity in
+  `CompactVectorBuilder::with_capacity` to pre-allocate bit storage.
+- Replaced `BitVectorBuilder::new` with `with_capacity` that allocates from an
+  `anybytes::ByteArea` section and plumbed `SectionWriter` through
+  `CompactVectorBuilder` and wavelet matrix builders.
+- Builders now track capacity and error when pushes exceed the reserved size.
 - Made `DacsByte` generic over its flag index type with a default of `Rank9SelIndex`.
 - `DacsByte::from_slice` now accepts a generic index type, removing `from_slice_with_index`.
 - Added `BitVectorBuilder` and zero-copy `BitVectorData` backed by `anybytes::View`.
@@ -12,11 +57,15 @@
 - Rename crate from `succdisk` to `jerky`.
 - Replaced the old `BitVector` with the generic `BitVector<I>` and renamed the
   mutable variant to `RawBitVector`.
-- Extended `BitVectorBuilder` with `push_bits` and `set_bit` APIs.
+- Replaced the push-based `BitVectorBuilder` with fixed-size `set_bit` and `set_bits` APIs and updated builders accordingly.
+- Added `set_bits_from_iter` to `BitVectorBuilder` and later revised it to take a
+  start offset and consume bits until the iterator ends or the builder is
+  full, leaving any unconsumed items to the caller.
 - Added `from_bit` constructor on `BitVectorBuilder` for repeating a single bit.
 - `DacsByte` now stores level data as zero-copy `View<[u8]>` values.
-- Added `to_bytes` and `from_bytes` on `DacsByte` for zero-copy serialization.
-- Documented the byte layout produced by `DacsByte::to_bytes` with ASCII art.
+- Replaced `to_bytes` helpers with `metadata` methods returning `SectionHandle`s
+  so structures can be reconstructed zero-copy via `from_bytes`.
+- Documented the byte layout for `DacsByte` sequences with ASCII art.
 - Switched `anybytes` dependency to track the upstream Git repository for the
   latest changes.
 - Removed internal byte buffers from data structures; `WaveletMatrix`,
@@ -31,13 +80,15 @@
 - `Rank9Sel` now stores a `BitVector<Rank9SelIndex>` built via `BitVectorBuilder`.
 - Replaced `DArrayFullIndex` with new `DArrayIndex` that uses const generics
   to optionally include `select1` and `select0` support.
-- Introduced `CompactVectorBuilder` mutable APIs `push_int`, `set_int`, and `extend`.
+- Introduced `CompactVectorBuilder` mutable APIs `set_int` and `set_ints`.
 - Simplified bit vector imports by re-exporting `BitVectorBuilder` and `Rank9SelIndex` and updating examples.
 - Moved the `bit_vector::bit_vector` module contents directly into `bit_vector` for cleaner paths.
+- Recorded future work items for a metadata serialization trait and
+  ByteArea-backed documentation examples.
 - Added README usage example demonstrating basic bit vector operations.
 - Removed `bit_vector::prelude`; import traits directly with `use jerky::bit_vector::*`.
 - Added `freeze()` on `CompactVectorBuilder` yielding an immutable `CompactVector` backed by `BitVector<NoIndex>`.
-- `CompactVector::new` and `with_capacity` now return builders; other constructors build via the builder pattern.
+- Removed `CompactVector::new`; use `with_capacity` to construct builders.
 - Wavelet matrix and DACs builders now use `BitVectorBuilder` for temporary bit
   vectors, storing only immutable `BitVector` data after construction.
 - Removed obsolete `RawBitVector` type.
@@ -66,3 +117,13 @@
 - Documented `WaveletMatrix` usage in `README.md`.
 - Moved README usage examples to runnable files in `examples/`.
 - Added `compact_vector` example showing construction and retrieval.
+- Serialized `WaveletMatrix` and `DacsByte` directly into a `ByteArea` to
+  avoid intermediate copies and guarantee contiguous layout.
+- Enabled doctests for `WaveletMatrix` by removing `ignore` fences from its
+  documentation examples.
+- `DacsByte::from_slice` now writes level bytes and flags directly into
+  `SectionWriter` buffers, removing the intermediate `Vec` allocations.
+- Stored per-level `DacsByte` handles in the byte arena, allowing
+  `DacsByteMeta` to reference a single handle slice like `WaveletMatrixMeta`.
+- Expanded examples and README with `ByteArea`/`SectionHandle` metadata
+  reconstruction for set-based APIs, adding a `dacs_byte` usage demo.

Cargo.toml

@@ -19,7 +19,7 @@ rust-version = "1.61.0"
 [dependencies]
 anyhow = "1.0"
 num-traits = "0.2.15"
-anybytes = { git = "https://github.com/triblespace/anybytes", features = ["zerocopy"] }
+anybytes = { git = "https://github.com/triblespace/anybytes", features = ["zerocopy", "mmap"] }
 zerocopy = "0.8"
 
 [features]

INVENTORY.md

@@ -9,9 +9,44 @@
 - Investigate alternative dense-select index strategies to replace removed `DArrayIndex`.
 - Explore additional index implementations leveraging the new generic `DacsByte<I>`.
 - Demonstrate the generic `from_slice` usage in examples and docs.
-- Showcase `DacsByte` byte serialization in an example.
-- Provide serialization helpers for additional structures beyond `WaveletMatrix`.
-- Show `CompactVector::to_bytes` and `from_bytes` in examples.
+- Apply `with_capacity` constructors across builders to avoid intermediate reallocations.
+- Transition builders to fixed-size APIs, removing growable variants.
+- Refactor builders and serializers to operate on `ByteArea` sections, enabling
+  zero-copy persistence across all structures.
+- Move `DacsByte` metadata arrays into the arena and store per-level handles
+  similar to `WaveletMatrixMeta`.
+- Add slice-based range setters for integer builders to minimize manual index
+  tracking during construction.
+- Provide bulk bit setters like `set_bits_from_slice` for `BitVectorBuilder`
+  to copy from packed data efficiently.
+- Provide convenience helpers to manage `ByteArea` and `SectionWriter` setup for
+  common builder use cases.
+- Audit remaining constructors for zero-capacity variants and decide whether to
+  offer explicit `empty` helpers instead of `with_capacity(0)`.
+- Allocate temporary wavelet-matrix buffers from `ByteArea` to avoid
+  intermediate `Vec` copies and ensure fully contiguous construction.
+- Provide a derive or macro to reduce boilerplate when implementing the
+  `Serializable` trait.
+- Consider a slice-based `WaveletMatrix` constructor to avoid requiring
+  cloneable iterators.
+ - Benchmark the cycle-based partitioning in `WaveletMatrixBuilder::freeze`
+   and explore more efficient permutation strategies.
+- Explore specialized rotation helpers for `BitVectorBuilder` to speed up
+  recursive partitioning without extra buffers.
+- Explore using `BitVectorBuilder` for other temporary bitmaps to reduce
+  scattered `Vec<bool>` allocations.
+- Review documentation examples across modules and convert remaining ignored
+  snippets into runnable doctests.
+- Explore iterating layer indices instead of reversing `remaining` to avoid
+  the upfront `reverse` cost in `WaveletMatrixBuilder::freeze`.
+- Audit integer-vector constructors for opportunities to allocate directly
+  from `SectionWriter` without temporary `Vec`s.
+- Document the fixed 64-bit word assumption across structures now that bit
+  vectors use `u64` internally.
+- Provide helpers on `SectionHandle` to derive typed sub-handles, reducing
+  manual offset math in complex `from_bytes` implementations like `DacsByte`.
+- Investigate slimming `DacsByte` per-level metadata to avoid storing unused
+  flag handles for the last level.
 
 ## Discovered Issues
 - `katex.html` performs manual string replacements; consider DOM-based manipulation.

README.md

@@ -25,48 +25,57 @@ RUSTDOCFLAGS="--html-in-header katex.html" cargo doc --no-deps
 ## Zero-copy bit vectors
 
 `BitVectorBuilder` can build a bit vector whose underlying `BitVectorData`
-is backed by `anybytes::View`. The data can be serialized with
-`BitVectorData::to_bytes` and reconstructed using `BitVectorData::from_bytes`,
-allowing zero-copy loading from an mmap or any other source by passing the
-byte region to `Bytes::from_source`.
+is backed by `anybytes::View`. Metadata describing a stored sequence includes
+[`SectionHandle`](anybytes::area::SectionHandle)s so the raw
+`Bytes` returned by `ByteArea::freeze` can be handed to
+`BitVectorData::from_bytes` for zero‑copy reconstruction.
 
-`DacsByte` sequences support a similar interface with `to_bytes` returning
-metadata alongside the byte slice and `from_bytes` rebuilding the sequence
-using that metadata.
+Types following this pattern implement the [`Serializable`](src/serialization.rs) trait,
+which exposes a `metadata` accessor and a `from_bytes` constructor.
+
+`DacsByte` sequences expose a `metadata` method returning a descriptor with a
+handle to a slice of per-level handles. Each entry stores the flag bitvector
+handle (if any), its bit length, and the payload byte handle. `from_bytes`
+rebuilds the sequence using that metadata.
 
 ```text
-Bytes layout from `DacsByte::to_bytes`:
+Bytes layout for a `DacsByte` sequence (current builders place sections
+contiguously, though layout is fully described by the stored handles):
 
 | flag[0] words | flag[1] words | ... | flag[n-2] words | level[0] data | level[1] data | ... | level[n-1] data |
 
 The flag vectors come first and store native-endian `usize` words. The level
 data immediately follows without any padding.
 ```
 
-`CompactVector` offers similar helpers: `CompactVector::to_bytes` returns a
-metadata struct along with the raw bytes, and `CompactVector::from_bytes`
-reconstructs the vector from that information.
-
-`WaveletMatrix` sequences share this layout and can be serialized with
-`WaveletMatrix::to_bytes` (returning metadata and bytes) and reconstructed
-using `WaveletMatrix::from_bytes`.
-
-The byte buffer returned by `to_bytes` stores each bit-vector layer
-contiguously. Given `num_words = ceil(len / WORD_LEN)`, the layout is:
-
-```
-bytes:
-+------------+------------+-----+
-| layer 0    | layer 1    | ... |
-| num_words  | num_words  |     |
-+------------+------------+-----+
+`CompactVector` and `WaveletMatrix` provide the same pattern: call `metadata`
+to obtain a descriptor with the required `SectionHandle`s, then hand both the
+metadata and the full `Bytes` region to `from_bytes`.
+
+For a wavelet matrix the metadata stores a handle to a slice of per-layer
+handles. Each handle in that slice points to the native-endian `usize` words
+forming a single layer. Layers may reside anywhere in the arena and no longer
+need to be contiguous.
+
+```rust
+use anybytes::ByteArea;
+use jerky::int_vectors::{CompactVector, CompactVectorBuilder};
+
+let mut area = ByteArea::new()?;
+let mut sections = area.sections();
+let mut builder = CompactVectorBuilder::with_capacity(3, 3, &mut sections)?;
+builder.set_ints(0..3, [7, 2, 5])?;
+let cv = builder.freeze();
+let meta = cv.metadata();
+let bytes = area.freeze()?;
+let view = CompactVector::from_bytes(meta, bytes.clone())?;
+assert_eq!(view.get_int(1), Some(2));
 ```
-where each segment contains `num_words` consecutive `usize` words for a layer.
 
 ## Examples
 
 See the [examples](examples/) directory for runnable usage demos, including
-`bit_vector`, `wavelet_matrix`, and `compact_vector`.
+`bit_vector`, `compact_vector`, `dacs_byte`, and `wavelet_matrix`.
 
 ## Licensing
 

bench/benches/timing_bitvec_rank.rs

@@ -7,6 +7,7 @@ use criterion::{
     criterion_group, criterion_main, measurement::WallTime, BenchmarkGroup, Criterion, SamplingMode,
 };
 
+use anybytes::ByteArea;
 use jerky::bit_vector::{BitVector, BitVectorBuilder, NoIndex, Rank, Rank9SelIndex};
 
 const SAMPLE_SIZE: usize = 30;
@@ -39,15 +40,23 @@ fn run_queries<R: Rank>(idx: &R, queries: &[usize]) {
 
 fn perform_bitvec_rank(group: &mut BenchmarkGroup<WallTime>, bits: &[bool], queries: &[usize]) {
     group.bench_function("jerky/BitVector", |b| {
-        let mut builder = BitVectorBuilder::new();
-        builder.extend_bits(bits.iter().cloned());
+        let mut area = ByteArea::new().unwrap();
+        let mut sections = area.sections();
+        let mut builder = BitVectorBuilder::with_capacity(bits.len(), &mut sections).unwrap();
+        for (i, &bval) in bits.iter().enumerate() {
+            builder.set_bit(i, bval).unwrap();
+        }
         let idx: BitVector<NoIndex> = builder.freeze();
         b.iter(|| run_queries(&idx, &queries));
     });
 
     group.bench_function("jerky/BitVector<Rank9SelIndex>", |b| {
-        let mut builder = BitVectorBuilder::new();
-        builder.extend_bits(bits.iter().cloned());
+        let mut area = ByteArea::new().unwrap();
+        let mut sections = area.sections();
+        let mut builder = BitVectorBuilder::with_capacity(bits.len(), &mut sections).unwrap();
+        for (i, &bval) in bits.iter().enumerate() {
+            builder.set_bit(i, bval).unwrap();
+        }
         let idx = builder.freeze::<Rank9SelIndex>();
         b.iter(|| run_queries(&idx, &queries));
     });

bench/benches/timing_bitvec_select.rs

@@ -3,6 +3,7 @@ use std::time::Duration;
 use rand::{Rng, SeedableRng};
 use rand_chacha::ChaChaRng;
 
+use anybytes::ByteArea;
 use criterion::{
     criterion_group, criterion_main, measurement::WallTime, BenchmarkGroup, Criterion, SamplingMode,
 };
@@ -42,15 +43,23 @@ fn run_queries<S: Select>(idx: &S, queries: &[usize]) {
 
 fn perform_bitvec_select(group: &mut BenchmarkGroup<WallTime>, bits: &[bool], queries: &[usize]) {
     group.bench_function("jerky/BitVector", |b| {
-        let mut builder = BitVectorBuilder::new();
-        builder.extend_bits(bits.iter().cloned());
+        let mut area = ByteArea::new().unwrap();
+        let mut sections = area.sections();
+        let mut builder = BitVectorBuilder::with_capacity(bits.len(), &mut sections).unwrap();
+        for (i, &bval) in bits.iter().enumerate() {
+            builder.set_bit(i, bval).unwrap();
+        }
         let idx: BitVector<NoIndex> = builder.freeze();
         b.iter(|| run_queries(&idx, &queries));
     });
 
     group.bench_function("jerky/BitVector<Rank9SelIndex>", |b| {
-        let mut builder = BitVectorBuilder::new();
-        builder.extend_bits(bits.iter().cloned());
+        let mut area = ByteArea::new().unwrap();
+        let mut sections = area.sections();
+        let mut builder = BitVectorBuilder::with_capacity(bits.len(), &mut sections).unwrap();
+        for (i, &bval) in bits.iter().enumerate() {
+            builder.set_bit(i, bval).unwrap();
+        }
         let idx = builder.freeze::<Rank9SelIndex>();
         b.iter(|| run_queries(&idx, &queries));
     });

bench/benches/timing_chrseq_access.rs

@@ -3,6 +3,7 @@ use std::time::Duration;
 use rand::{Rng, SeedableRng};
 use rand_chacha::ChaChaRng;
 
+use anybytes::ByteArea;
 use jerky::bit_vector::*;
 use jerky::char_sequences::WaveletMatrix;
 use jerky::int_vectors::CompactVector;
@@ -25,8 +26,9 @@ const PROTEINS_PSEF_STR: &str = include_str!("../data/texts/proteins.1MiB.txt");
 // In effective alphabet
 fn load_text(s: &str) -> CompactVector {
     let mut text = s.as_bytes().to_vec();
-    let mut builder = BitVectorBuilder::new();
-    builder.extend_bits(core::iter::repeat(false).take(256));
+    let mut area = ByteArea::new().unwrap();
+    let mut sections = area.sections();
+    let mut builder = BitVectorBuilder::with_capacity(256, &mut sections).unwrap();
     for &c in &text {
         builder.set_bit(c as usize, true).unwrap();
     }
@@ -92,7 +94,11 @@ fn perform_chrseq_access(group: &mut BenchmarkGroup<WallTime>, text: &CompactVec
     let queries = gen_random_ints(NUM_QUERIES, 0, text.len(), SEED_QUERIES);
 
     group.bench_function("jerky/WaveletMatrix<Rank9SelIndex>", |b| {
-        let idx = WaveletMatrix::<Rank9SelIndex>::new(text.clone()).unwrap();
+        let alph_size = text.iter().max().unwrap() + 1;
+        let mut area = ByteArea::new().unwrap();
+        let mut sections = area.sections();
+        let idx = WaveletMatrix::<Rank9SelIndex>::from_iter(alph_size, text.iter(), &mut sections)
+            .unwrap();
         b.iter(|| run_queries(&idx, &queries));
     });
 }

bench/benches/timing_intvec_access.rs

@@ -7,6 +7,7 @@ use criterion::{
     criterion_group, criterion_main, measurement::WallTime, BenchmarkGroup, Criterion, SamplingMode,
 };
 
+use anybytes::ByteArea;
 use jerky::int_vectors::Access;
 
 const SAMPLE_SIZE: usize = 30;
@@ -87,7 +88,10 @@ fn perform_intvec_access(group: &mut BenchmarkGroup<WallTime>, vals: &[u32]) {
     });
 
     group.bench_function("jerky/DacsByte", |b| {
-        let idx = jerky::int_vectors::DacsByte::from_slice(vals).unwrap();
+        let mut area = ByteArea::new().unwrap();
+        let mut writer = area.sections();
+        let idx = jerky::int_vectors::DacsByte::from_slice(vals, &mut writer).unwrap();
+        area.freeze().unwrap();
         b.iter(|| run_queries(&idx, &queries));
     });
 }