Decode u64s rather than u8s (#78)

* Add from_u64s and decode_u64s for panic-free, alignment-check-free decoding

Adds a new decode path that preserves u64 alignment information through
the entire pipeline, eliminating per-field alignment checks that
bytemuck::try_cast_slice required when going through &[u8].

Key changes:
- decode_u64s: returns (&[u64], u8) pairs instead of &[u8] slices,
  where the u8 indicates valid trailing bytes in the last word
- from_u64s on FromBytes: non-panicking field construction that enables
  LLVM to eliminate unused tuple fields as dead code
- validate/validate_typed: upfront structural and type-compatibility
  checks for encoded data, replacing the implicit panic-on-bad-data
- Remove inspect module (superseded by examples/decode_asm.rs)

Assembly impact for accessing field 0 of a k-tuple of u64s:
  Old (from_bytes): k=3 → 133 insns, k=8 → 273 insns (linear in k)
  New (from_u64s):  k=3 → 68 insns,  k=8 → 68 insns  (constant in k)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Remove EncodeDecode trait and Sequence encoding, rename module to indexed

Indexed is now the sole encoding format with inherent methods, so callers
don't need to import a trait. The Sequence format provided no random
access or u64-aligned decoding and is no longer needed.

Renames serialization_neu to indexed now that there is no other
serialization module to distinguish from.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Move validate_typed to FromBytes::validate, remove Indexed struct

validate_typed was partly a function of the Indexed format and partly
of the type being decoded. It now lives as FromBytes::validate, which
combines structural and type-compatibility checks using element_sizes.

The Indexed struct's methods were all one-line delegates to free
functions in the indexed module. Removed the struct and inlined
length_in_words/length_in_bytes as free functions. Callers use the
module directly (columnar::bytes::indexed::encode, etc).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Hide element_sizes as implementation detail of FromBytes::validate

element_sizes is only used internally by validate. Mark it #[doc(hidden)]
and simplify tests to exercise validate directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Make both element_sizes and validate public on FromBytes

element_sizes is public for implementors to override.
validate is public for callers to use at trust boundaries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Clean up decode_asm example to three representative approaches

Trimmed from experimental accumulation to a clean comparison of:
- from_bytes + decode (O(k) baseline)
- from_u64s + decode_u64s (O(1) in k via dead code elimination)
- decode_field random access (O(1) in both k and field position)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Update benchmarks to use indexed module instead of removed Sequence

The bench and serde benchmarks referenced the removed EncodeDecode
trait and Sequence type. Updated to use the indexed module directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix from_u64s for Discriminant and updated enum container layout

The enum container struct now uses a single `indexes: Discriminant`
field instead of separate `variant` and `offset` fields. Update the
derive macro's from_u64s to match, and add from_u64s/element_sizes
to the Discriminant FromBytes impl.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Rework validate to take decoded slices, add validate_typed entry point

FromBytes::validate now takes &[(&[u64], u8)] matching the from_u64s
input shape, making it composable for nested types. Added
indexed::validate_typed::<T> as the single entry point that combines
structural and type-level validation. Also added from_u64s and
element_sizes for Discriminant.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Rename validate to validate_structure, validate_typed to validate

The obvious name should do the obvious thing: indexed::validate::<T>
does full validation (structural + type compatibility). The structural-
only check is now validate_structure, an implementation detail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Remove unused FromBytes import in test module

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: frankmcsherry

benches/bench.rs

@@ -1,6 +1,6 @@
 use bencher::{benchmark_group, benchmark_main, Bencher};
 use columnar::{Clear, Columnar};
-use columnar::bytes::{EncodeDecode, Sequence};
+use columnar::bytes::indexed;
 
 fn empty_copy(bencher: &mut Bencher) { _bench_copy(bencher, vec![(); 1024]); }
 fn option_copy(bencher: &mut Bencher) { _bench_copy(bencher, vec![Option::<String>::None; 1024]); }
@@ -61,7 +61,7 @@ fn _bench_copy<T: Columnar+Eq>(bencher: &mut Bencher, record: T) where T::Contai
         arena.push(&record);
     }
     use columnar::Borrow;
-    bencher.bytes = Sequence::length_in_bytes(&arena.borrow()) as u64;
+    bencher.bytes = indexed::length_in_bytes(&arena.borrow()) as u64;
     arena.clear();
 
     bencher.iter(|| {
@@ -83,7 +83,7 @@ fn _bench_extend<T: Columnar+Eq>(bencher: &mut Bencher, record: T) where T::Cont
         arena.push(&record);
     }
     use columnar::{Borrow, Container};
-    bencher.bytes = Sequence::length_in_bytes(&arena.borrow()) as u64;
+    bencher.bytes = indexed::length_in_bytes(&arena.borrow()) as u64;
 
     let arena2 = arena.clone();
 

benches/serde.rs

@@ -1,6 +1,6 @@
 use bencher::{benchmark_group, benchmark_main, Bencher};
 use columnar::{Columnar, Container, Clear, FromBytes};
-use columnar::bytes::{EncodeDecode, Sequence};
+use columnar::bytes::indexed;
 use serde::{Serialize, Deserialize};
 
 fn goser_new(b: &mut Bencher) {
@@ -19,7 +19,7 @@ fn goser_push(b: &mut Bencher) {
         container.push(&log);
     }
     let mut words = vec![];
-    Sequence::encode(&mut words, &container.borrow());
+    indexed::encode(&mut words, &container.borrow());
     b.bytes = 8 * words.len() as u64;
     b.iter(|| {
         container.clear();
@@ -50,11 +50,11 @@ fn goser_encode(b: &mut Bencher) {
         container.push(&log);
     }
     let mut words = vec![];
-    Sequence::encode(&mut words, &container.borrow());
+    indexed::encode(&mut words, &container.borrow());
     b.bytes = 8 * words.len() as u64;
     b.iter(|| {
         words.clear();
-        Sequence::encode(&mut words, &container.borrow());
+        indexed::encode(&mut words, &container.borrow());
         bencher::black_box(&words);
     });
 }
@@ -67,10 +67,10 @@ fn goser_decode(b: &mut Bencher) {
     for _ in 0..1024 {
         container.push(&log);
     }
-    Sequence::encode(&mut words, &container.borrow());
+    indexed::encode(&mut words, &container.borrow());
     b.bytes = 8 * words.len() as u64;
     b.iter(|| {
-        let mut slices = Sequence::decode(&mut words);
+        let mut slices = indexed::decode(&mut words);
         let foo = <<Log as Columnar>::Container as Container>::Borrowed::from_bytes(&mut slices);
         bencher::black_box(foo);
     });

columnar_derive/src/lib.rs

@@ -331,6 +331,10 @@ fn derive_struct(name: &syn::Ident, generics: &syn::Generics, data_struct: syn::
                     )*
                     Self { #(#names,)* }
                 }
+                #[inline(always)]
+                fn from_u64s(words: &mut impl Iterator<Item=(&'columnar [u64], u8)>) -> Self {
+                    Self { #(#names: ::columnar::FromBytes::from_u64s(words),)* }
+                }
             }
         }
     };
@@ -519,6 +523,11 @@ fn derive_unit_struct(name: &syn::Ident, _generics: &syn::Generics, vis: syn::Vi
             fn from_byte_slices(bytes: &[&'columnar [u8]]) -> Self {
                 Self { count: &::columnar::bytemuck::try_cast_slice(bytes[0]).unwrap()[0] }
             }
+            #[inline(always)]
+            fn from_u64s(words: &mut impl Iterator<Item=(&'columnar [u64], u8)>) -> Self {
+                let (w, _tail) = words.next().expect("Iterator exhausted prematurely");
+                Self { count: &w[0] }
+            }
         }
 
         impl ::columnar::Columnar for #name {
@@ -910,6 +919,13 @@ fn derive_enum(name: &syn::Ident, generics: &syn:: Generics, data_enum: syn::Dat
                     let indexes = <::columnar::Discriminant<CVar, COff, CC>>::from_byte_slices(&bytes[_offset ..]);
                     Self { #(#names,)* indexes }
                 }
+                #[inline(always)]
+                fn from_u64s(words: &mut impl Iterator<Item=(&'columnar [u64], u8)>) -> Self {
+                    Self {
+                        #(#names: ::columnar::FromBytes::from_u64s(words),)*
+                        indexes: ::columnar::FromBytes::from_u64s(words),
+                    }
+                }
             }
         }
     };
@@ -1203,6 +1219,10 @@ fn derive_tags(name: &syn::Ident, _generics: &syn:: Generics, data_enum: syn::Da
             fn from_byte_slices(bytes: &[&'columnar [u8]]) -> Self {
                 Self { variant: CVar::from_byte_slices(bytes) }
             }
+            #[inline(always)]
+            fn from_u64s(words: &mut impl Iterator<Item=(&'columnar [u64], u8)>) -> Self {
+                Self { variant: ::columnar::FromBytes::from_u64s(words) }
+            }
         }
 
         impl ::columnar::Columnar for #name {

examples/decode_asm.rs

@@ -0,0 +1,99 @@
+//! Assembly inspection for decode paths.
+//!
+//! Compares three approaches to accessing a single field of a k-tuple
+//! stored in Indexed-encoded `&[u64]` data:
+//!
+//! 1. `from_bytes` + `decode`: constructs all k fields, O(k)
+//! 2. `from_u64s` + `decode_u64s`: non-panicking, LLVM eliminates unused fields, O(1) in k
+//! 3. `decode_field` (random access): decodes one field directly, O(1) in k and j
+//!
+//! Build with: `cargo rustc --example decode_asm --release -- --emit asm`
+
+use columnar::*;
+use columnar::bytes::indexed;
+
+// ================================================================
+// from_bytes path (construct all k fields, access field j)
+// ================================================================
+
+#[no_mangle] pub fn bytes_3_f0(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64]);
+    T::from_bytes(&mut indexed::decode(store)).0[i]
+}
+#[no_mangle] pub fn bytes_3_flast(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64]);
+    T::from_bytes(&mut indexed::decode(store)).2[i]
+}
+#[no_mangle] pub fn bytes_8_f0(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64], &'a [u64],
+                   &'a [u64], &'a [u64], &'a [u64], &'a [u64]);
+    T::from_bytes(&mut indexed::decode(store)).0[i]
+}
+#[no_mangle] pub fn bytes_8_flast(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64], &'a [u64],
+                   &'a [u64], &'a [u64], &'a [u64], &'a [u64]);
+    T::from_bytes(&mut indexed::decode(store)).7[i]
+}
+
+// ================================================================
+// from_u64s path (non-panicking, LLVM eliminates unused fields)
+// ================================================================
+
+#[no_mangle] pub fn u64s_3_f0(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64]);
+    T::from_u64s(&mut indexed::decode_u64s(store)).0[i]
+}
+#[no_mangle] pub fn u64s_3_flast(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64]);
+    T::from_u64s(&mut indexed::decode_u64s(store)).2[i]
+}
+#[no_mangle] pub fn u64s_8_f0(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64], &'a [u64],
+                   &'a [u64], &'a [u64], &'a [u64], &'a [u64]);
+    T::from_u64s(&mut indexed::decode_u64s(store)).0[i]
+}
+#[no_mangle] pub fn u64s_8_flast(store: &[u64], i: usize) -> u64 {
+    type T<'a> = (&'a [u64], &'a [u64], &'a [u64], &'a [u64],
+                   &'a [u64], &'a [u64], &'a [u64], &'a [u64]);
+    T::from_u64s(&mut indexed::decode_u64s(store)).7[i]
+}
+
+// ================================================================
+// Random access (decode one field directly, O(1) in both k and j)
+// ================================================================
+
+/// Decode field `k` directly from store as `(&[u64], u8)`.
+/// Each call is independent — no iterator state.
+#[inline(always)]
+fn decode_field(store: &[u64], k: usize) -> (&[u64], u8) {
+    let slices = store[0] as usize / 8 - 1;
+    let index = &store[..slices + 1];
+    let last = *index.last().unwrap_or(&0) as usize;
+    let last_w = (last + 7) / 8;
+    let words = &store[..last_w];
+    let upper = (*index.get(k + 1).unwrap_or(&0) as usize).min(last);
+    let lower = (((*index.get(k).unwrap_or(&0) as usize) + 7) & !7).min(upper);
+    let upper_w = ((upper + 7) / 8).min(words.len());
+    let lower_w = (lower / 8).min(upper_w);
+    let tail = (upper % 8) as u8;
+    (&words[lower_w..upper_w], tail)
+}
+
+#[no_mangle] pub fn field_3_f0(store: &[u64], i: usize) -> u64 {
+    decode_field(store, 0).0[i]
+}
+#[no_mangle] pub fn field_3_flast(store: &[u64], i: usize) -> u64 {
+    decode_field(store, 2).0[i]
+}
+#[no_mangle] pub fn field_8_f0(store: &[u64], i: usize) -> u64 {
+    decode_field(store, 0).0[i]
+}
+#[no_mangle] pub fn field_8_flast(store: &[u64], i: usize) -> u64 {
+    decode_field(store, 7).0[i]
+}
+
+fn main() {
+    let mut store = vec![0u64; 100];
+    store[0] = 32; store[1] = 32; store[2] = 32; store[3] = 32;
+    println!("{}", std::hint::black_box(field_3_f0(std::hint::black_box(&store), 0)));
+}