🌪 0.54.0

Changes

⚠️ Breaks

• chore: remove merge's explicit nullability (#4967) @danking
• chore[btrblocks]: move validity in deltas & enabled delta encoding (#4905) @joseph-isaacs

✨ Features

• feat: teach constant to fill_null (#4985) @danking
• Feature: Make ListView the canonical encoding for List (#4946) @connortsui20
• feat: GithubArchive benchmark for nested data (#4960) @a10y
• feat: collect patch offsets per chunk during alp encode (#4965) @0ax1
• Support sum stat for decimal arrays (#4963) @AdamGS
• feat: take_search constant time index lookups (#4951) @0ax1
• Add BitBuffer to vortex-buffer (#4937) @gatesn
• feat: constant time patches slicing (#4913) @0ax1
• Feature: add lance to TPC-H benchmark (#4919) @connortsui20
• Feature: Add lance to clickbench benchmarks (#4908) @connortsui20
• Feature: Add lance compress benchmark (#4907) @connortsui20
• feat: Allow read from stream (#4906) @m7kss1
• Fix: Implement missing duckdb to vortex type conversions (#4841) @robert3005
• RLE gpu decompress kernel (#4864) @robert3005
• feat: add concat expression to vortex-expr (#4902) @blaginin
• Feature: Add Lance to our random access benchmarks (#4882) @connortsui20
• feat[gpu]: for kernel and framework for running pipeline (#4857) @joseph-isaacs
• feat: auto-enable tsan for sanitizer builds (#4849) @0ax1
• feat: wire in fastlanes rle to btrblocks (#4789) @0ax1
• feat: teach Python about (part of) VortexWriteOptions (#4825) @danking
• Use the original DataFusion provided predicate to dynamically prune files (#4836) @AdamGS
• feat[gpu]: dict masked take kernel (#4839) @joseph-isaacs
• feat[gpu]: dict take kernel (#4833) @joseph-isaacs
• Expand expression pushdown to DataFusion (#4398) @AdamGS
• feat: FineWeb as a benchmark dataset (#4828) @a10y
• Fastlanes GPU bit unpack kernel (#4820) @robert3005
• Feature: add listview rebuild + cleanup (#4819) @connortsui20
• Initial version of Masked encoding (#4785) @AdamGS
• feat[dtype]: Rename and move ScalarType ->NativeDType, scalar -> dtype (#4793) @joseph-isaacs
• compact() compacts recursively (#4790) @onursatici
• feat[fuzz]: support slow comparison using make_comparator for fallback arrow compare (#4783) @joseph-isaacs
• Feature: ListViewBuilder (#4738) @connortsui20
• Feature: ListView Compute Functions (#4712) @connortsui20

🚀 Performance

• perf[fastlanes]: fused for bitpacking decode (#4955) @joseph-isaacs
• Use arrow's bit iterators for BitBuffer (#4952) @gatesn
• chore[btrblocks]: move validity in deltas & enabled delta encoding (#4905) @joseph-isaacs
• Perf: Create splits via indices instead of layout reader (#4891) @connortsui20
• perf[gpu]: fused AoT for and bitpacking kernel (#4872) @joseph-isaacs
• cheaper StructArray::clone (#4877) @onursatici
• field_by_name to use indices lookup (#4868) @onursatici
• faster StructFields::find (#4863) @onursatici
• varbinview compact to zero copy whenever possible (#4827) @onursatici
• Use DataFusion's FilePruner to prune files based on stats and partition values (#4831) @AdamGS
• perf[duckdb]: use a single external data/vector buffer per exporter (#4826) @joseph-isaacs
• perf[array]: don't delete filters (#4823) @joseph-isaacs
• perf[duckdb]: zero-copy decimal exporter (#4812) @joseph-isaacs
• perf[duckdb]: zero-copy primitive exporter (#4804) @joseph-isaacs
• Use from_trusted_len_iter when applying take on views buffer (#4807) @AdamGS
• Implement filter for SequenceArray (#4799) @AdamGS
• Pre-allocate data buffer in VarBin::take (#4749) @AdamGS

🐛 Bug Fixes

27 changes

• fix[vortex-io]: register read_at intent in InstrumentedReadAt (#4991) @asubiotto
• Support prunning on the newly supported NullCount (#4990) @AdamGS
• fix[vortex-expr]: implement stat_falsification for IsNullExpr (#4980) @asubiotto
• fix: Continue to prefer List over ListView by default in ToArrow (#4982) @a10y
• fix[vortex-expr]: implement stat_falsification for BetweenExpr (#4977) @asubiotto
• Coerce and cast decimal sum (#4968) @AdamGS
• Fix: lance feature flag for clickbench nightly (#4976) @connortsui20
• chore: change 4 lines; eliminate countless Arc clones (#4972) @danking
• fix[btr-blocks]: cfg(unstable_encodings) imports (#4957) @joseph-isaacs
• fix: Builder ensure_capacity is now reserve_exact to acquire additional and not total capacity (#4962) @robert3005
• Fix: ensure capacity bug in list builder (#4956) @connortsui20
• Fix: temporarily fix Arrow FixedSizeList bug (#4953) @connortsui20
• Fix the return dtype of arithmetic binary expressions (#4950) @AdamGS
• fix: match_between rewriting (#4945) @a10y
• fix[vortex-{scan,io,datafusion}]: propagate tracing spans (#4939) @asubiotto
• Fix: revert Lance v2.1->v2.0 for compression benchmark (#4936) @connortsui20
• Fix: fix summaries when missing (#4933) @connortsui20
• Release the GIL before block_on (#4931) @gatesn
• Propegate output_ordering to scan (#4909) @AdamGS
• fix[gpu]: u8, u16, u64 bitpack (#4850) @joseph-isaacs
• fix: only use SparseScheme if we can cascade (#4840) @0ax1
• binary view debug impl (#4832) @onursatici
• fix[encodings]: fix visit_children on sliced PCO/ZSTD encoded arrays (#4813) @asubiotto
• fix[duckdb]: handle dict and null correctly (#4788) @joseph-isaacs
• Zip Fragmentation on take (#4747) @blaginin
• fix[duckdb]: handle count(*) queries correctly using an EMPTY column (#4765) @joseph-isaacs
• fix[array]: nested Operator format fix (#4746) @joseph-isaacs

🧰 Maintenance

112 changes

• FIx java bindings packaging (#5008) @AdamGS
• chore: avoid copy when building chunk_offsets in chunked layout reader (#5006) @robert3005
• Split the publish flow into two flows (#5004) @AdamGS
• do not export spans by default on benchmarks (#5003) @onursatici
• Move fastlanes back to released version (#4999) @AdamGS
• Small correction to pre-commit script (#4998) @AdamGS
• Dont publish vortex-gpu-kernels (#4997) @AdamGS
• chore(deps): update all patch updates (#4995) @renovate[bot]
• Stop enabling pyo3/extension-modules by default (#4992) @AdamGS
• Chore: reorganize compute function files in vortex-array (#4984) @connortsui20
• fix: run gharchive on develop (#4981) @a10y
• chore(deps): lock file maintenance (#4925) @renovate[bot]
• chore: Only run cuda tests with cuda feature (#4975) @robert3005
• feat[gpu]: jit kernels (#4920) @joseph-isaacs
• chore[ci]: order the target of bench runs datafusion > duckdb (#4970) @joseph-isaacs
• chore: Enable zstd feature for fuzzer to read files compressed with compact compressor (#4948) @robert3005
• chore[fuzz]: fuzz vortex compact (file_io) (#4947) @joseph-isaacs
• chore: Fuzzer can use builtin compare for nested types (#4944) @robert3005
• Chore: feature flag lance (#4938) @connortsui20
• chore[fuzz]: fuzz vortex compact (#4943) @joseph-isaacs
• Chore: Clean up benchmarks website (#4921) @connortsui20
• Chore: bump lance to version 2.0->2.1 (#4930) @connortsui20
• Try and reduce renovate noise (#4929) @AdamGS
• Update Rust crate datafusion to v50.2.0 (#4927) @renovate[bot]
• Update Rust crate datafusion-catalog to v50.2.0 (#4928) @renovate[bot]
• Update Rust crate regex to v1.12.1 (#4922) @renovate[bot]
• Chore: rename benchmark charts in compression group (#4918) @connortsui20
• chore[ci] faster tpch sf=0.1 (#4917) @joseph-isaacs
• chore[array]: downcast -> narrow (#4916) @joseph-isaacs
• Don't require field name copy for expression partitioning (#4912) @robert3005
• Dont publish vortex-gpu (#4911) @AdamGS
• Update Rust crate zip to v6 (#4904) @renovate[bot]
• chore(deps): update rust crate cc to v1.2.41 (#4910) @renovate[bot]
• Update Rust crate half to v2.7.0 (#4900) @renovate[bot]
• Chore: move public BI benchmark script to scripts (#4897) @connortsui20
• chore[gpu] fls-gen -> vortex-gpu-kernels (#4896) @joseph-isaacs
• perf[gpu]: fls gen refactor (#4894) @joseph-isaacs
• Chore: rename lance random access benchmark on website (#4893) @connortsui20
• fix(deps): update dependency org.immutables:value to v2.11.6 (#4892) @renovate[bot]
• Return filter from FileSource (#4890) @AdamGS
• Chore: bump codspeed back up to v4 (#4887) @connortsui20
• Really stop python-version from beign changed by renovate (#4886) @AdamGS
• chore(deps): update dependency python to 3.14 (#4880) @renovate[bot]
• Yet another attempt at stopping automated python version updates (#4885) @AdamGS
• Stop renovate from updating python docker image version (#4884) @AdamGS
• chore(deps): update rust crate criterion to 0.7 (#4876) @renovate[bot]
• chore(deps): update fsfe/reuse-action action to v6 (#4852) @renovate[bot]
• fix(deps): update all patch updates (patch) (#4883) @renovate[bot]
• chore[gpu]: test dict take (+ float) (#4881) @joseph-isaacs
• Chore: clean up random access benchmark (#4878) @connortsui20
• chore(deps): update python docker tag (#4879) @renovate[bot]
• chore(deps): update all patch updates (patch) (#4855) @renovate[bot]
• Revert "chore(deps): update python docker tag" (#4870) @robert3005
• chore(deps): update python docker tag (#4869) @renovate[bot]
• Minor simplification of the serde dependency (#4866) @AdamGS
• fix(deps): update rust crate fastlanes to 0.4.0 (#4805) @renovate[bot]
• wider tables in compress bench (#4867) @onursatici
• Chore: make list and list view builders consistent (#4861) @connortsui20
• chore: drop unused patches search_sorted fn (#4865) @0ax1
• pack expr bench (#4862) @onursatici
• Chore: Revert codspeed to 3.0.5 (#4859) @connortsui20
• Chore: Make validate on arrays public (#4858) @connortsui20
• chore(deps): update crate-ci/typos action to v1.38.0 (#4856) @renovate[bot]
• chore(deps): update rust crate divan to v4 (#4838) @renovate[bot]
• actually (#4846) @connortsui20
• chore(deps): update all patch updates (patch) (#4845) @renovate[bot]
• Chore: Remove some TODOs (#4843) @connortsui20
• chore: Simplify stats truncation, optimise take function stats inheritance (#4835) @robert3005
• Chore: vortex-array modules cleanup (#4818) @connortsui20
• revert divan codspeed compat to v3 (#4834) @onursatici
• chore(deps): update rust crate divan to v4 (#4829) @renovate[bot]
• Upgrade to DF 50.1.0 (#4787) @AdamGS
• Feature: add listview rebuild + cleanup (#4819) @connortsui20
• ci: add helpful remarks to filter benchmark results (#4824) @a10y
• varbinview compact micro bench (#4821) @onursatici
• chore(deps): update all patch updates (patch) (#4817) @renovate[bot]
• chore: MaskedArray simplification for nullable validity (#4815) @robert3005
• chore: MaskedArray requires a nullable validity (#4810) @robert3005
• chore[bench]: add --explain and --explain-analyze flags (#4811) @joseph-isaacs
• chore: zip struct per field (#4797) @blaginin
• Chore: change OffsetPType to IntegerPType and update NativePType (#4780) @connortsui20
• [chore] update README.md with more citations (#4801) @lwwmanning
• chore(deps): update gradle/actions action to v5 (#4806) @renovate[bot]
• Remove unused GH workflow (#4802) @AdamGS
• chore(deps): update crate-ci/typos action to v1.37.0 (#4800) @renovate[bot]
• Implement filter for SequenceArray (#4799) @AdamGS
• fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.19 (#4798) @renovate[bot]
• Fix typo in parca config (#4796) @AdamGS
• Update parca agent in CI (#4795) @AdamGS
• fix(deps): update junit-framework monorepo to v5.14.0 (minor) (#4791) @renovate[bot]
• Propegate panic from tokio async code correctly (#4786) @AdamGS
• chore(deps): update all patch updates (patch) (#4781) @renovate[bot]
• chore(deps): lock file maintenance (#4721) @renovate[bot]
• chore(deps): update all patch updates (patch) (#4775) @renovate[bot]
• chore(deps): update opentelemetry-rust monorepo to 0.31.0 (minor) (#4773) @renovate[bot]
• disallow changing builder size with set_validity (#4766) @onursatici
• chore(deps): update plugin com.diffplug.spotless to v8 (#4752) @renovate[bot]
• chore: enable test_vortex_scan_ultra_deep_nesting on ci (#4774) @0ax1
• chore(deps): update crate-ci/typos action to v1.36.3 (#4772) @renovate[bot]
• Revert "chore: switch to metal instance for codspeed benchmarks" (#4768) @robert3005
• Display predicate SQL in datafusion TreeRender (#4769) @AdamGS
• Check typos in CI and fix existing ones (#4764) @AdamGS
• Bug: DuckDB List Validation assertion failure (#4753) @connortsui20
• chore(deps): update all patch updates (patch) (#4763) @renovate[bot]
• chore(deps): update spiraldb/actions action to v0.18.0 (#4762) @renovate[bot]
• Unpin nextest version (#4761) @AdamGS
• fix(deps): update all patch updates to v1.0.186 (patch) (#4756) @renovate[bot]
• fix(deps): update dependency org.immutables:value to v2.11.4 (#4755) @renovate[bot]
• Pipeline row selection (#4740) @gatesn
• Improve varbin take benchmarks (#4751) @AdamGS
• chore(deps): update rust crate wasm-bindgen-futures to v0.4.54 (#4748) @renovate[bot]
• chore: bump duckdb to 1.4.0 (#4667) @0ax1