Skip to content

Commit 858ff9b

Browse files
gatesnrobert3005
andauthored
Fix unaligned metadata flake (#2067)
Rkyv uses aligned access by default, but our metadata flatbufferis &[u8], therefore has no alignment requirements. The unaligned feature is marginally slower (unaligned load instructions), but handles this case. We can use aligned access again when we move to 8-byte metadata and store it as a u64 in flatbuffer. --------- Co-authored-by: Robert Kruszewski <[email protected]>
1 parent 1d36132 commit 858ff9b

File tree

5 files changed

+5
-4
lines changed

5 files changed

+5
-4
lines changed

Cargo.toml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,8 @@ rand_distr = "0.4"
121121
rayon = "1.10.0"
122122
regex = "1.11.0"
123123
reqwest = { version = "0.12.0", features = ["blocking"] }
124-
rkyv = { version = "0.8", features = ["little_endian", "pointer_width_32", "bytecheck"] }
124+
# TODO(ngates): remove the unaligned rkyv feature when we move to 8-byte metadata.
125+
rkyv = { version = "0.8", features = ["little_endian", "pointer_width_32", "bytecheck", "unaligned"] }
125126
rstest = "0.24"
126127
serde = "1.0.197"
127128
serde_json = "1.0.116"

docs/quickstart.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ Vortex array:
3535
>>> parquet = pq.read_table("_static/example.parquet")
3636
>>> vtx = vortex.array(parquet)
3737
>>> vtx.nbytes
38-
141057
38+
141055
3939

4040
Compress
4141
^^^^^^^^
@@ -46,7 +46,7 @@ Use :func:`~vortex.encoding.compress` to compress the Vortex array and check the
4646

4747
>>> cvtx = vortex.compress(vtx)
4848
>>> cvtx.nbytes
49-
15888
49+
15755
5050
>>> cvtx.nbytes / vtx.nbytes
5151
0.11...
5252

-6 Bytes
Binary file not shown.
-13 Bytes
Binary file not shown.

vortex-sampling-compressor/tests/smoketest.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,7 @@ mod tests {
168168
assert_eq!(chunk.encoding().id(), VarBinEncoding::ID);
169169
assert_eq!(
170170
chunk.statistics().get(Stat::UncompressedSizeInBytes),
171-
Some(Scalar::from(134357009_u64))
171+
Some(Scalar::from(134357007_u64))
172172
);
173173
}
174174

0 commit comments

Comments
 (0)