🌪 0.55.0

Changes

• Fix: remove TPC-H SF 1000 from nightly benchmarks (#5235) @connortsui20

⚠️ Breaks

• Expression VTable (#5191) @gatesn
• Vortex Session (#5111) @gatesn

✨ Features

• Feature: add in-place filter for BufferMut + benchmarks (#5238) @connortsui20
• Feature: add into_mut to vectors (#5221) @connortsui20
• Feature: add into_mut methods on mask and buffer (#5214) @connortsui20
• feat: OperatorVTable::bind for VarBin/VarBinView (#5212) @a10y
• Vector scalars (#5213) @gatesn
• pickling support (#5011) @onursatici
• Feature: add Decimal::bind (#5210) @connortsui20
• Bind constant array (#5208) @gatesn
• feat: error instead of panic on too many buffers (#5162) @danking
• Feature: Add naive ListView::bind (#5207) @connortsui20
• Feature: ListVector (#5190) @connortsui20
• feat[scalar]: zero_value (#5181) @joseph-isaacs
• Chore: add immutable validity for mutable vectors (#5188) @connortsui20
• chore[array]: fill_null cleanup + decimal impl (#5175) @joseph-isaacs
• feat[vector]: fill in VectorMut::with_capacity (#5173) @joseph-isaacs
• Feature: Support importing nested vectors as well as timestamp ns (#5178) @robert3005
• Feature: FixedSizeListVector (#5127) @connortsui20
• feat: rest of BinaryViewVector{,Mut} (#5133) @a10y
• feat: compact when the buffers are empty or if more than u16::MAX (#5166) @danking
• Worker pool for current thread runtimes (#5134) @gatesn
• feat: into_struct_fields_* (#5128) @danking
• Add decimal vectors (#5124) @gatesn
• VarBinVector (#5115) @gatesn
• Array operator optimizer (#5113) @gatesn
• Feature: Conversion from PrimitiveVector to PrimitiveArray (#5114) @connortsui20
• GPU Scan (#5012) @robert3005
• Adds a GetItem operator array (#5110) @gatesn
• Operator trait for sequence array (#5108) @gatesn
• MaskedArray + masked compute + is_null/is_not_null (#5104) @gatesn
• Feature: add Struct OperatorVTable::bind (#5105) @connortsui20
• Add Decimal type to dtypes documentation (#5107) @a10y
• Vector comparison compute (#5100) @gatesn
• Add checked arithmetic compute (#5096) @gatesn
• Feature: Struct Vectors (#5094) @connortsui20
• feat: write nested struct layouts (#4942) @a10y
• Feature: Add IntoIter implementation for vectors (#5093) @connortsui20
• Feature: Add methods to PVectorMut similar to std::vec::Vec (#5079) @connortsui20
• feat[layout]: support array encoding outlined in flat layout (#5086) @joseph-isaacs
• feat[layout]: dict layout known code nullability (#5063) @joseph-isaacs
• Array Operator API (#5022) @gatesn
• Vortex vector compute (#5054) @gatesn
• feat: add some more examples in vortex crate (#5019) @a10y
• Add ArrayHash and ArrayEq traits (#5009) @gatesn
• Use BitBuffer instead of BooleanBuffer (#4940) @robert3005
• feat: teach merge to optionally error on duplicates (#4971) @danking
• Feature: Add vortex-vector (#5010) @connortsui20

🚀 Performance

• Feature: add in-place filter for BufferMut + benchmarks (#5238) @connortsui20
• Benchmarks: make random access timed + median instead of set iterations (#5234) @connortsui20
• experiment: eagerly decode dict on read (#5229) @a10y
• perf: fuse alp decode with applying patches (#5065) @0ax1
• Performance: ListViewArray zero-copyable to ListArray optimizations (#5129) @connortsui20
• perf[btrblocks]: write list (not list view). (#5087) @joseph-isaacs
• performance[vortex-datafusion]: support pushing down filters on struct columns (#5024) @asubiotto

🐛 Bug Fixes

39 changes

• Fix: casting Zstd array to non nullable requires decompression (#5258) @robert3005
• Fix: silence codspeed warning (#5253) @connortsui20
• fix[fuzz]: apply mask to validity in baseline (#5249) @joseph-isaacs
• Fix: make Filter compute take self instead of &self (#5248) @connortsui20
• fix[array]: take propagate is_constant if indices are all_valid (#5241) @joseph-isaacs
• fix[pco]: cast condition check (#5239) @joseph-isaacs
• Do not do eager formatting of error strings (#5233) @gatesn
• fix: patch_chunk offset calc (#5222) @0ax1
• fix: Remove erroneous chunk offset calculation from Patches::take_search (#5220) @robert3005
• fix[array]: min_max of nan is none constant array (#5200) @joseph-isaacs
• fix[array]: min_max return type non nullable (#5202) @joseph-isaacs
• Fix: merge error (#5216) @connortsui20
• Do not construct backtrace for opt (#5198) @gatesn
• chore: rename PValue::as_primitive to cast (#5194) @joseph-isaacs
• fix[pco]: cast array with validity (#5193) @joseph-isaacs
• fix[array]: handle empty/all_invalid sum correctly (#5180) @joseph-isaacs
• fix[vortex-expr]: delegate AnalysisExpr to CastExpr child (#5179) @asubiotto
• fix[array]: validity mask into nullable (#5171) @joseph-isaacs
• fix[fuzz]: correct chunked mask nullable (#5170) @joseph-isaacs
• fix[fuzz]: correct list take nullable baseline (#5169) @joseph-isaacs
• fix: Handle nulls in bool array min/max (#5163) @robert3005
• chore[array]: fsl append null scalar (#5156) @joseph-isaacs
• chore[fuzzer]: fixup idx selection scalar_at (#5154) @joseph-isaacs
• Fix: incorrect fuzz slice for FixedSizeList (#5155) @connortsui20
• fix[array]: chunked array sum int correctly (#5147) @joseph-isaacs
• fix[fuzz]: mask baseline dtype from validity (#5145) @joseph-isaacs
• fix[array]: fill_null don't run utf8 or binary (#5151) @joseph-isaacs
• fix[array]: invalid arbitrary decimal scalar value (#5143) @joseph-isaacs
• fix[fuzz]: scalar_at baseline handle nulls (#5150) @joseph-isaacs
• fix[array]: compare using arrow with different physical types (#5142) @joseph-isaacs
• Fix: revert List::take compute function (#5099) @connortsui20
• fix: fix conformance tests, fix delta cast, impl constant minmax (#5101) @danking
• fix[ci]: polar signals profile perm (#5084) @joseph-isaacs
• fix: patch chunk offsets for alp encode (#5080) @0ax1
• Fix: incorrect FromIterator<bool> implementation (#5056) @connortsui20
• change gharchive query to reduce memory blowup, disable Delta (#5026) @a10y
• fix[decimal]: add sum support for decimals to chunked and constant (#5023) @joseph-isaacs
• fix[ci]: allow fork PRs to run benchmarks (#5001) @joseph-isaacs
• fix: include offset in fastlanes rle metadata (#5015) @0ax1

🧰 Maintenance

88 changes

• [DataFusion] Don't pushdown empty filters (#5264) @AdamGS
• Bump DF to 50.3.0 (#5263) @AdamGS
• chore: Fix fuzzers baseline nullability for compare and take (#5255) @robert3005
• Bump codspeed (action and dependency) (#5261) @AdamGS
• chore: Fix fuzzers baseline nullability for decimal fill_null (#5257) @robert3005
• Update all patch updates (#5259) @renovate[bot]
• Update crate-ci/typos action to v1.39.0 (#5260) @renovate[bot]
• Chore: remove feature gates from vortex-compute (#5254) @connortsui20
• Batch execution API (#5242) @gatesn
• Add conversions from Vectors into ArrayRef (#5243) @gatesn
• chore: Convert DuckDB vectors directly to vortex vectors (#5228) @robert3005
• chore: Slightly nicer fuzzer assertions (#5240) @robert3005
• chore: fix semantic merge conflict (#5232) @a10y
• chore: trivial error message fix (#5231) @a10y
• Chore: bitpacking move stuff around again (#5224) @connortsui20
• Chore: add vortex vector and compute dependencies to encodings/fastlanes (#5211) @connortsui20
• validate object store response length (#5219) @onursatici
• Remove unused scope (#5217) @gatesn
• chore[benchmark-site]: incremental jsonl decompression/parsing (#5203) @lwwmanning
• Chore: use Self in more places in vectors (#5204) @connortsui20
• Chore: remove the sized bound on vector trait impl (#5199) @connortsui20
• Chore: require all vectors to be sized (#5197) @connortsui20
• chore: BinaryViewVector tests and validation fix (#5182) @a10y
• chore: deny let_underscore_drop (#5095) @danking
• Chore: Reorganize documentation and modules for vectors (#5186) @connortsui20
• Remove concat expression (#5183) @gatesn
• Chore: add docs and tests for decimal vectors (#5185) @connortsui20
• chore[vector]: run fmt (#5172) @joseph-isaacs
• Chore: Decimal vector cleanup (#5165) @connortsui20
• Chore: use actual type names in vortex-vector (#5167) @connortsui20
• chore[docs]: document committers and maintainers in CONTRIBUTING (#5164) @lwwmanning
• Remove old operator API from vortex-expr (#5160) @gatesn
• Chore: use try_new for struct array into_arrow (#5159) @connortsui20
• chore[fuzzer]: disable fsl for now (#5153) @joseph-isaacs
• chore(deps): update all patch updates (#5139) @renovate[bot]
• chore[array]: decimal builder use decimal dtype (#5148) @joseph-isaacs
• chore[fuzz]: run fuzz in rel with debug remove multiple jobs (#5149) @joseph-isaacs
• chore[dtype]: use NonZero for decimal precision (#5146) @joseph-isaacs
• Update junit-framework monorepo to v5.14.1 (#5141) @renovate[bot]
• Update dependency ch.qos.logback:logback-classic to v1.5.20 (#5140) @renovate[bot]
• chore: quiet vortex_btrblocks (#5130) @danking
• Chore: clean up struct vector (#5126) @connortsui20
• Remove mimalloc from vortex-python (#5125) @AdamGS
• Chore: Add match_vector_pair macro (#5123) @connortsui20
• Revert "Update CodSpeedHQ/action digest to 4348f63" (#5122) @robert3005
• Update CodSpeedHQ/action digest to 4348f63 (#5082) @renovate[bot]
• Move and rename DecimalType (#5120) @gatesn
• Chore: move fastlanes bitpacking modules and files around (#5116) @connortsui20
• Chore: un-ignore macro doc tests (#5112) @connortsui20
• Update all patch updates (#5083) @renovate[bot]
• Add operator for null and extension arrays (#5098) @gatesn
• chore: use vx array assert eq macro 2 (#5077) @joseph-isaacs
• PrimitiveArray operator vtable (#5091) @gatesn
• Move Array BatchKernel to sync API (#5090) @gatesn
• Chore: fix from_iter impls in vectors (#5075) @connortsui20
• Exact layout reader row count (#5088) @gatesn
• chore[encodings]: use vx array assert eq macro (#5076) @joseph-isaacs
• chore[vx]: more decompression throughput benchmarks (#5067) @joseph-isaacs
• chore[fuzz]: use vx array assert eq macro (#5073) @joseph-isaacs
• Chore: Move stuff around in vectors (#5072) @connortsui20
• chore[duckdb]: set max threads (#5062) @joseph-isaacs
• chore: Map buffer to new output benchmark (#5071) @robert3005
• Chore: add more vector constructors (#5070) @connortsui20
• chore[fuzz]: add fill_null baselinne (#5061) @joseph-isaacs
• Chore: Clean up extend impls for BufferMut (#5055) @connortsui20
• Pin codpeed action (#5064) @AdamGS
• chore[fuzz]: add mask_fn baseline (#5047) @joseph-isaacs
• Fix benchmark units (#5057) @AdamGS
• chore[ci]: benchmarks to take the median of runs not the min (#5052) @joseph-isaacs
• Simplify the serde implementation for StructFields (#5046) @AdamGS
• fix[ci]: more cat-s3 retries and sleep a little between tries (#5051) @joseph-isaacs
• chore: use assert_arrays_eq in more places (#4988) @danking
• Additions to vector API (#5045) @gatesn
• chore[fuzz]: add scalar_at action (#5043) @joseph-isaacs
• chore: bump duckdb to 1.4.1 (#5041) @0ax1
• Chore: Add more docs + from_option_iter (#5031) @connortsui20
• Insta is dev dep (#5037) @blaginin
• chore[fuzz]: add new compute function fuzzing [fill_null, mask, max/min, sum] (#5036) @joseph-isaacs
• chore[tui]: tree display layout (#5035) @joseph-isaacs
• chore[ci]: remove unnecessarly workflow perms. (#5034) @joseph-isaacs
• chore[array]: tree display no /0 (#5032) @joseph-isaacs
• chore[docs]: update fmt doc (#5002) @joseph-isaacs
• Chore: Remove nullability from Vectors (#5029) @connortsui20
• Use pin-project-lite (#5021) @gatesn
• chore[duckdb]: set max threads (#5020) @joseph-isaacs
• chore: ZonedLayoutReader children are lazy (#5007) @robert3005
• fix: bring back StructArray::into_record_batch_with_schema (#5018) @a10y
• chore[gpu]: feature flag gpu tests (#5016) @joseph-isaacs