🌪 0.53.0

Changes

• Update bug_report.md (#4524) @robert3005
• Create bug report template (#4523) @robert3005
• Revert "Allow bench results to collapse in GH comment" (#4516) @gatesn

⚠️ Breaks

• Refactor VortexWrite to return Stream and Footer (#4557) @gatesn

✨ Features

• Zero copy from any Vec into a Buffer (#4742) @AdamGS
• Operator Updates (#4725) @gatesn
• Feature: ListViewArray (#4708) @connortsui20
• Add serde feature to buffers and expose it externally (#4709) @AdamGS
• Rename functions in SelectExpr to make API clearer (#4690) @AdamGS
• feat: allow for linking DuckDB debug build (#4705) @0ax1
• Reapply "feat: take for sequence array" (#4674) (#4685) @robert3005
• feat: take for sequence array (#4654) @danking
• feat: fastlanes rle (#4588) @0ax1
• Bump DataFusion to 50 and arrow to 56 (#4577) @AdamGS
• feat: FoRArray is sorted if its compressed form is sorted (#4652) @danking
• feat: teach DeserializeStep to Debug (#4649) @danking
• Move scan to use handles (#4646) @gatesn
• buffered bytes counter in layout strategy (#4656) @onursatici
• feat: is_sorted for sequence array (#4653) @danking
• Explicitly register wall kernels in compute vtables in benchmarks (#4650) @AdamGS
• Feature: Implement FixedSizeList in PyVortex bindings (#4618) @connortsui20
• Move VortexRead over to new File I/O (#4626) @gatesn
• Create a SharedSegmentCache for re-using existing futures (#4640) @gatesn
• Support hive-style partitioning in DF (#4638) @AdamGS
• Add spawn_blocking to runtime (#4623) @gatesn
• Push-based write API (#4608) @gatesn
• Support independent footer serde (#4598) @gatesn
• Feature: Add DuckDB conversion for fixed-size lists / ARRAY type (#4590) @connortsui20
• Add back segment paths to the inspect command (#4600) @gatesn
• Refactor VortexWrite to return Stream and Footer (#4557) @gatesn
• DictionaryEncoder to store nulls in values (#4596) @gatesn
• Make FieldName an actual type an not just an alias (#4575) @AdamGS
• feat: teach vx.array how to convert Python range into SequenceArray (#4571) @danking
• feat: teach vortex-compact to compress decimal arrays (#4569) @danking
• Feature: Implementation of ConstantArray and SparseArray canonicalize for FixedSizeList (#4545) @connortsui20
• File I/O abstraction for new runtimes (#4540) @gatesn
• Feature: Add FixedSizeList fuzzing support (#4541) @connortsui20
• feat: New vx inspect tool (#4499) @AdamGS
• Feature: Implement Compute Functions for FixedSizeListArray (#4495) @connortsui20
• Add a Vortex runtime abstraction (#4520) @gatesn
• feat: teach vortex-tui to show full name paths (#4529) @danking
• feat: Constant compression for strings (#4512) @a10y

🚀 Performance

• Buffer more data in object store writer (#4619) @AdamGS
• exponential buffer growth for varbinview builder (#4701) @onursatici
• Inline some scalar code (#4644) @AdamGS
• Avoid wastefully allocating Vecs in StructScalar::partial_cmp (#4609) @AdamGS
• row range scans to prune unneeded splits (#4602) @onursatici
• performance: Avoid intersecting ranges multiple times in scan (#4578) @robert3005
• use the footer on count_rows where possible (#4567) @onursatici
• Remove evaluations, use MaskFuture (#4494) @gatesn

🐛 Bug Fixes

31 changes

• fix: Casting primitive nullable values doesn't cast invalid values (#4735) @robert3005
• Fix: FixedSizeList DuckDB exporter bug (#4610) @connortsui20
• Fix: UncompressedSizeInBytes calculates recursive uncompressed size (#4693) @robert3005
• fix: duckdb parquet metadata option (#4689) @0ax1
• fix: zero sized duckdb array (#4688) @0ax1
• fix: duckdb test type mismatch (#4687) @0ax1
• Fix: dict bool canonical preserves nullability (#4684) @robert3005
• Fix: restore and correct dict compare where only single value matches the search (#4606) @robert3005
• fix: duckdb dict ownership in export (#4681) @0ax1
• fix: fastlanes rle decode validity (#4675) @0ax1
• Remove local path optimization for object store (#4673) @gatesn
• Revert "feat: take for sequence array" (#4674) @AdamGS
• Fix: Don't assume date time parts arrays are unsigned (#4630) @robert3005
• Convert handles to hold weak reference to their executor (#4624) @gatesn
• Shutdown Tokio file (#4621) @gatesn
• Fix: write_tokio was renamed to write, fix conflict (#4616) @robert3005
• Fix: simplify dict array compute functions and report correct dictarray dtype now that codes and values don't have same nullability (#4579) @robert3005
• Remove transmutes from vortex-io/runtime (#4572) @gatesn
• Fix: Don't pushdown binary numeric operations into dict arrays (#4560) @robert3005
• Fix: DecimalArray#sum doesn't panic (#4565) @robert3005
• VarBinArray#slice doesn't slice bytes (#4561) @robert3005
• fix: is_sorted returns nullable bool (#4555) @danking
• fix: default to no explicit setting of cache_metadata (#4550) @danking
• Fix: IsSortedKernel can return unknown, fix IsSorted for DictArray (#4546) @robert3005
• fix[duckdb]: handle dict with constant value with validity exporter (#4544) @joseph-isaacs
• fix: Correctly handle slicing RunEnd arrays where slice end aligns with run end (#4542) @robert3005
• cancelled sequence futures should remove their wakers (#4536) @onursatici
• fix: fix deserialization bug in pco (#4530) @danking
• fix: teach zstd to include correct validity when decompressing utf8 (#4511) @danking
• fix: teach vortex-duckdb to safely copy validity (#4534) @danking
• fix: #4503 (#4504) @danking

🧰 Maintenance

99 changes

• Add benchmarks for StructArray scalar_at and take (#4737) @AdamGS
• chore(deps): update rust crate flatbuffers to v25.9.23 (#4744) @renovate[bot]
• chore(deps): update plugin com.gradleup.shadow to v9.2.1 (#4743) @renovate[bot]
• expose row_block_size in WriteStrategy (#4741) @onursatici
• fix(deps): update all patch updates to v3.5.7 (patch) (#4739) @renovate[bot]
• Minimal docs for vortex-btrblocks (#4730) @AdamGS
• chore(deps): update rust crate moka to v0.12.11 (#4733) @renovate[bot]
• chore: switch to metal instance for codspeed benchmarks (#4732) @0ax1
• chore(deps): update rust crate object_store to v0.12.4 (#4731) @renovate[bot]
• chore[tui]: fuzzy search column names (#4729) @joseph-isaacs
• Make ArrayStreamAdapter only accept relevant stream (#4694) @AdamGS
• chore: clarify ninja dep error message (#4724) @0ax1
• chore(deps): update rust crate tempfile to v3.23.0 (#4728) @renovate[bot]
• chore: clarify row count assertion (#4723) @0ax1
• chore: drop invalid sql test as it triggers exception breakpoints (#4722) @0ax1
• chore(deps): update rust crate wasm-bindgen-futures to v0.4.53 (#4716) @renovate[bot]
• chore(deps): update rust crate serde to v1.0.226 (#4720) @renovate[bot]
• chore: use DuckDB debug build in Rust tests (sanitizer) (#4718) @0ax1
• Chore: Some small test fixes (#4713) @connortsui20
• chore(deps): update rust crate clap to v4.5.48 (#4714) @renovate[bot]
• chore: run vortex-duckdb tests in ci (#4711) @0ax1
• Chore: rename some methods (#4710) @connortsui20
• chore(deps): update rust crate primitive-types to 0.14.0 (#4703) @renovate[bot]
• chore(deps): update rust crate anyhow to v1.0.100 (#4706) @renovate[bot]
• chore: drop "invalid sql" test as it triggers exception breakpoints (#4702) @0ax1
• chore(deps): update rust crate cc to v1.2.38 (#4700) @renovate[bot]
• fix(deps): update all patch updates to v1.0.185 (patch) (#4698) @renovate[bot]
• Update pyproject.toml to use dependency-groups (#4696) @AdamGS
• only canonicalise before compression in repartition writer (#4692) @onursatici
• docs: clarify duckdb dictionary and dictionary slice (#4691) @0ax1
• coalescing metrics (#4666) @onursatici
• dict layout options to have u16 max size (#4657) @onursatici
• fix(deps): update dependency com.google.guava:guava to v33.5.0-jre (#4686) @renovate[bot]
• remove unused dispatcher (#4682) @onursatici
• chore(deps): update rust crate wasm-bindgen-futures to v0.4.53 (#4680) @renovate[bot]
• chore(deps): update rust crate wasm-bindgen-futures to v0.4.52 (#4671) @renovate[bot]
• chore(deps): update rust crate serde to v1.0.225 (#4663) @renovate[bot]
• chore(deps): update rust crate fastlanes to v0.2.2 (#4655) @renovate[bot]
• chore(deps): update rust crate pyo3 to 0.26.0 (#4443) @renovate[bot]
• Chore: add uv sync --all-packages in check.sh (#4617) @connortsui20
• chore(deps): update rust crate serde to v1.0.224 (#4645) @renovate[bot]
• Minor ScaleValue refactoring (#4643) @AdamGS
• VortexReadAt to return static futures (#4639) @gatesn
• Updates to Vortex I/O (#4634) @gatesn
• chore(deps): lock file maintenance (#4633) @renovate[bot]
• chore(deps): update rust crate serde to v1.0.223 (#4631) @renovate[bot]
• fix(deps): update all patch updates (patch) (#4632) @renovate[bot]
• chore(deps): update rust crate serde to v1.0.221 (#4629) @renovate[bot]
• chore(deps): update rust crate serde to v1.0.220 (#4628) @renovate[bot]
• Remove compio (#4622) @gatesn
• chore: expose DeserializeStep (#4614) @danking
• chore: buffer! when possible (#4594) @danking
• chore: add r/place and airquality datasets to benchmark (#4613) @danking
• fix(deps): update all patch updates (patch) (#4612) @renovate[bot]
• fix(deps): update protobuf monorepo to v4.32.1 (patch) (#4607) @renovate[bot]
• Chore: Add no-op IsSorted and MinMax compute kernels for FixedSizeListVTable (#4601) @connortsui20
• fix(deps): update all patch updates (patch) (#4604) @renovate[bot]
• Fix typo in arrays documentation (#4603) @evmar
• Build docs with all features in docs.rs (#4597) @AdamGS
• Chore: Add missing FixedSizeListEncoding (#4592) @connortsui20
• chore(deps): update rust crate zip to v5.1.0 (#4595) @renovate[bot]
• chore: allow casting empty all invalid arrays to non-nullable (#4462) @blaginin
• Add results to vortex-bench (#4580) @gatesn
• Chore: Specialize append_scalar and remove append_option (#4591) @connortsui20
• chore(deps): update spiraldb/actions action to v0.16.0 (#4589) @renovate[bot]
• fix(deps): update all patch updates (patch) (#4586) @renovate[bot]
• chore(deps): update rust crate tempfile to v3.22.0 (#4584) @renovate[bot]
• fix(deps): update all patch updates (patch) (#4583) @renovate[bot]
• Fix: simplify dict array compute functions and report correct dictarray dtype now that codes and values don't have same nullability (#4579) @robert3005
• Remove layout strategy generics (#4576) @gatesn
• Chore: Fix list builder bug (#4574) @connortsui20
• fix(deps): update all patch updates to v1.0.180 (patch) (#4573) @renovate[bot]
• chore(deps): update rust crate enum-iterator to v2.3.0 (#4570) @renovate[bot]
• chore(deps): update rust crate fastlanes to v0.2.1 (#4566) @renovate[bot]
• chore(deps): update rust crate chrono to v0.4.42 (#4563) @renovate[bot]
• chore(deps): lock file maintenance (#4562) @renovate[bot]
• fix(deps): update all patch updates to v1.0.179 (patch) (#4559) @renovate[bot]
• fix(deps): update all patch updates to v1.0.178 (patch) (#4558) @renovate[bot]
• fix(deps): update all patch updates to v1.0.177 (patch) (#4556) @renovate[bot]
• chore(deps): update rust crate compio to 0.16 (#4554) @renovate[bot]
• fix(deps): update all patch updates to v1.0.176 (patch) (#4553) @renovate[bot]
• chore(deps): update rust crate zip to v5 (#4551) @renovate[bot]
• Add more unchecked constructors (#4527) @connortsui20
• chore: enable vortex-compact in every benchmark (#4531) @danking
• chore(deps): update rust crate cc to v1.2.36 (#4538) @renovate[bot]
• Consolidate on futures crate (#4537) @gatesn
• Capitalize some words in the perf script (#4519) @AdamGS
• chore(deps): update rust crate wasm-bindgen-futures to v0.4.51 (#4533) @renovate[bot]
• chore(deps): update rust crate insta to v1.43.2 (#4532) @renovate[bot]
• Rename bug_report.md to bug_report.yml (#4526) @robert3005
• chore[ci]: natural change benchmark github post (#4518) @joseph-isaacs
• chore[ci]: make perf script less wordy (#4517) @joseph-isaacs
• chore(deps): update actions/setup-python action to v6 (#4515) @renovate[bot]
• fix(deps): update all patch updates to v1.0.175 (patch) (#4514) @renovate[bot]
• chore(deps): update aws-actions/configure-aws-credentials action to v5 (#4505) @renovate[bot]
• Allow bench results to collapse in GH comment (#4509) @gatesn
• chore(deps): update rust crate log to v0.4.28 (#4510) @renovate[bot]
• chore[ci]: tpch-10 on ci action (#4498) @joseph-isaacs
• chore[mask]: move mask future into vortex-array (#4496) @joseph-isaacs