🌪 0.53.0
·
723 commits
to develop
since this release
Changes
- Update bug_report.md (#4524) @robert3005
- Create bug report template (#4523) @robert3005
- Revert "Allow bench results to collapse in GH comment" (#4516) @gatesn
⚠️ Breaks
✨ Features
- Zero copy from any Vec into a Buffer (#4742) @AdamGS
- Operator Updates (#4725) @gatesn
- Feature:
ListViewArray(#4708) @connortsui20 - Add serde feature to buffers and expose it externally (#4709) @AdamGS
- Rename functions in SelectExpr to make API clearer (#4690) @AdamGS
- feat: allow for linking DuckDB debug build (#4705) @0ax1
- Reapply "feat: take for sequence array" (#4674) (#4685) @robert3005
- feat: take for sequence array (#4654) @danking
- feat: fastlanes rle (#4588) @0ax1
- Bump DataFusion to 50 and arrow to 56 (#4577) @AdamGS
- feat: FoRArray is sorted if its compressed form is sorted (#4652) @danking
- feat: teach DeserializeStep to Debug (#4649) @danking
- Move scan to use handles (#4646) @gatesn
- buffered bytes counter in layout strategy (#4656) @onursatici
- feat: is_sorted for sequence array (#4653) @danking
- Explicitly register wall kernels in compute vtables in benchmarks (#4650) @AdamGS
- Feature: Implement
FixedSizeListinPyVortexbindings (#4618) @connortsui20 - Move VortexRead over to new File I/O (#4626) @gatesn
- Create a SharedSegmentCache for re-using existing futures (#4640) @gatesn
- Support hive-style partitioning in DF (#4638) @AdamGS
- Add spawn_blocking to runtime (#4623) @gatesn
- Push-based write API (#4608) @gatesn
- Support independent footer serde (#4598) @gatesn
- Feature: Add DuckDB conversion for fixed-size lists / ARRAY type (#4590) @connortsui20
- Add back segment paths to the inspect command (#4600) @gatesn
- Refactor VortexWrite to return Stream and Footer (#4557) @gatesn
- DictionaryEncoder to store nulls in values (#4596) @gatesn
- Make
FieldNamean actual type an not just an alias (#4575) @AdamGS - feat: teach vx.array how to convert Python range into SequenceArray (#4571) @danking
- feat: teach vortex-compact to compress decimal arrays (#4569) @danking
- Feature: Implementation of
ConstantArrayandSparseArraycanonicalize forFixedSizeList(#4545) @connortsui20 - File I/O abstraction for new runtimes (#4540) @gatesn
- Feature: Add
FixedSizeListfuzzing support (#4541) @connortsui20 - feat: New vx inspect tool (#4499) @AdamGS
- Feature: Implement Compute Functions for
FixedSizeListArray(#4495) @connortsui20 - Add a Vortex runtime abstraction (#4520) @gatesn
- feat: teach vortex-tui to show full name paths (#4529) @danking
- feat: Constant compression for strings (#4512) @a10y
🚀 Performance
- Buffer more data in object store writer (#4619) @AdamGS
- exponential buffer growth for varbinview builder (#4701) @onursatici
- Inline some scalar code (#4644) @AdamGS
- Avoid wastefully allocating Vecs in StructScalar::partial_cmp (#4609) @AdamGS
- row range scans to prune unneeded splits (#4602) @onursatici
- performance: Avoid intersecting ranges multiple times in scan (#4578) @robert3005
- use the footer on count_rows where possible (#4567) @onursatici
- Remove evaluations, use MaskFuture (#4494) @gatesn
🐛 Bug Fixes
31 changes
- fix: Casting primitive nullable values doesn't cast invalid values (#4735) @robert3005
- Fix:
FixedSizeListDuckDB exporter bug (#4610) @connortsui20 - Fix: UncompressedSizeInBytes calculates recursive uncompressed size (#4693) @robert3005
- fix: duckdb parquet metadata option (#4689) @0ax1
- fix: zero sized duckdb array (#4688) @0ax1
- fix: duckdb test type mismatch (#4687) @0ax1
- Fix: dict bool canonical preserves nullability (#4684) @robert3005
- Fix: restore and correct dict compare where only single value matches the search (#4606) @robert3005
- fix: duckdb dict ownership in
export(#4681) @0ax1 - fix: fastlanes rle decode validity (#4675) @0ax1
- Remove local path optimization for object store (#4673) @gatesn
- Revert "feat: take for sequence array" (#4674) @AdamGS
- Fix: Don't assume date time parts arrays are unsigned (#4630) @robert3005
- Convert handles to hold weak reference to their executor (#4624) @gatesn
- Shutdown Tokio file (#4621) @gatesn
- Fix: write_tokio was renamed to write, fix conflict (#4616) @robert3005
- Fix: simplify dict array compute functions and report correct dictarray dtype now that codes and values don't have same nullability (#4579) @robert3005
- Remove transmutes from vortex-io/runtime (#4572) @gatesn
- Fix: Don't pushdown binary numeric operations into dict arrays (#4560) @robert3005
- Fix: DecimalArray#sum doesn't panic (#4565) @robert3005
- VarBinArray#slice doesn't slice bytes (#4561) @robert3005
- fix: is_sorted returns nullable bool (#4555) @danking
- fix: default to no explicit setting of cache_metadata (#4550) @danking
- Fix: IsSortedKernel can return unknown, fix IsSorted for DictArray (#4546) @robert3005
- fix[duckdb]: handle dict with constant value with validity exporter (#4544) @joseph-isaacs
- fix: Correctly handle slicing RunEnd arrays where slice end aligns with run end (#4542) @robert3005
- cancelled sequence futures should remove their wakers (#4536) @onursatici
- fix: fix deserialization bug in pco (#4530) @danking
- fix: teach zstd to include correct validity when decompressing utf8 (#4511) @danking
- fix: teach vortex-duckdb to safely copy validity (#4534) @danking
- fix: #4503 (#4504) @danking
🧰 Maintenance
99 changes
- Add benchmarks for StructArray scalar_at and take (#4737) @AdamGS
- chore(deps): update rust crate flatbuffers to v25.9.23 (#4744) @renovate[bot]
- chore(deps): update plugin com.gradleup.shadow to v9.2.1 (#4743) @renovate[bot]
- expose row_block_size in WriteStrategy (#4741) @onursatici
- fix(deps): update all patch updates to v3.5.7 (patch) (#4739) @renovate[bot]
- Minimal docs for vortex-btrblocks (#4730) @AdamGS
- chore(deps): update rust crate moka to v0.12.11 (#4733) @renovate[bot]
- chore: switch to metal instance for codspeed benchmarks (#4732) @0ax1
- chore(deps): update rust crate object_store to v0.12.4 (#4731) @renovate[bot]
- chore[tui]: fuzzy search column names (#4729) @joseph-isaacs
- Make
ArrayStreamAdapteronly accept relevant stream (#4694) @AdamGS - chore: clarify ninja dep error message (#4724) @0ax1
- chore(deps): update rust crate tempfile to v3.23.0 (#4728) @renovate[bot]
- chore: clarify row count assertion (#4723) @0ax1
- chore: drop invalid sql test as it triggers exception breakpoints (#4722) @0ax1
- chore(deps): update rust crate wasm-bindgen-futures to v0.4.53 (#4716) @renovate[bot]
- chore(deps): update rust crate serde to v1.0.226 (#4720) @renovate[bot]
- chore: use DuckDB debug build in
Rust tests (sanitizer)(#4718) @0ax1 - Chore: Some small test fixes (#4713) @connortsui20
- chore(deps): update rust crate clap to v4.5.48 (#4714) @renovate[bot]
- chore: run vortex-duckdb tests in ci (#4711) @0ax1
- Chore: rename some methods (#4710) @connortsui20
- chore(deps): update rust crate primitive-types to 0.14.0 (#4703) @renovate[bot]
- chore(deps): update rust crate anyhow to v1.0.100 (#4706) @renovate[bot]
- chore: drop "invalid sql" test as it triggers exception breakpoints (#4702) @0ax1
- chore(deps): update rust crate cc to v1.2.38 (#4700) @renovate[bot]
- fix(deps): update all patch updates to v1.0.185 (patch) (#4698) @renovate[bot]
- Update pyproject.toml to use dependency-groups (#4696) @AdamGS
- only canonicalise before compression in repartition writer (#4692) @onursatici
- docs: clarify duckdb dictionary and dictionary slice (#4691) @0ax1
- coalescing metrics (#4666) @onursatici
- dict layout options to have u16 max size (#4657) @onursatici
- fix(deps): update dependency com.google.guava:guava to v33.5.0-jre (#4686) @renovate[bot]
- remove unused dispatcher (#4682) @onursatici
- chore(deps): update rust crate wasm-bindgen-futures to v0.4.53 (#4680) @renovate[bot]
- chore(deps): update rust crate wasm-bindgen-futures to v0.4.52 (#4671) @renovate[bot]
- chore(deps): update rust crate serde to v1.0.225 (#4663) @renovate[bot]
- chore(deps): update rust crate fastlanes to v0.2.2 (#4655) @renovate[bot]
- chore(deps): update rust crate pyo3 to 0.26.0 (#4443) @renovate[bot]
- Chore: add
uv sync --all-packagesincheck.sh(#4617) @connortsui20 - chore(deps): update rust crate serde to v1.0.224 (#4645) @renovate[bot]
- Minor ScaleValue refactoring (#4643) @AdamGS
- VortexReadAt to return static futures (#4639) @gatesn
- Updates to Vortex I/O (#4634) @gatesn
- chore(deps): lock file maintenance (#4633) @renovate[bot]
- chore(deps): update rust crate serde to v1.0.223 (#4631) @renovate[bot]
- fix(deps): update all patch updates (patch) (#4632) @renovate[bot]
- chore(deps): update rust crate serde to v1.0.221 (#4629) @renovate[bot]
- chore(deps): update rust crate serde to v1.0.220 (#4628) @renovate[bot]
- Remove compio (#4622) @gatesn
- chore: expose DeserializeStep (#4614) @danking
- chore: buffer! when possible (#4594) @danking
- chore: add r/place and airquality datasets to benchmark (#4613) @danking
- fix(deps): update all patch updates (patch) (#4612) @renovate[bot]
- fix(deps): update protobuf monorepo to v4.32.1 (patch) (#4607) @renovate[bot]
- Chore: Add no-op
IsSortedandMinMaxcompute kernels forFixedSizeListVTable(#4601) @connortsui20 - fix(deps): update all patch updates (patch) (#4604) @renovate[bot]
- Fix typo in arrays documentation (#4603) @evmar
- Build docs with all features in docs.rs (#4597) @AdamGS
- Chore: Add missing
FixedSizeListEncoding(#4592) @connortsui20 - chore(deps): update rust crate zip to v5.1.0 (#4595) @renovate[bot]
- chore: allow casting empty all invalid arrays to non-nullable (#4462) @blaginin
- Add results to vortex-bench (#4580) @gatesn
- Chore: Specialize
append_scalarand removeappend_option(#4591) @connortsui20 - chore(deps): update spiraldb/actions action to v0.16.0 (#4589) @renovate[bot]
- fix(deps): update all patch updates (patch) (#4586) @renovate[bot]
- chore(deps): update rust crate tempfile to v3.22.0 (#4584) @renovate[bot]
- fix(deps): update all patch updates (patch) (#4583) @renovate[bot]
- Fix: simplify dict array compute functions and report correct dictarray dtype now that codes and values don't have same nullability (#4579) @robert3005
- Remove layout strategy generics (#4576) @gatesn
- Chore: Fix list builder bug (#4574) @connortsui20
- fix(deps): update all patch updates to v1.0.180 (patch) (#4573) @renovate[bot]
- chore(deps): update rust crate enum-iterator to v2.3.0 (#4570) @renovate[bot]
- chore(deps): update rust crate fastlanes to v0.2.1 (#4566) @renovate[bot]
- chore(deps): update rust crate chrono to v0.4.42 (#4563) @renovate[bot]
- chore(deps): lock file maintenance (#4562) @renovate[bot]
- fix(deps): update all patch updates to v1.0.179 (patch) (#4559) @renovate[bot]
- fix(deps): update all patch updates to v1.0.178 (patch) (#4558) @renovate[bot]
- fix(deps): update all patch updates to v1.0.177 (patch) (#4556) @renovate[bot]
- chore(deps): update rust crate compio to 0.16 (#4554) @renovate[bot]
- fix(deps): update all patch updates to v1.0.176 (patch) (#4553) @renovate[bot]
- chore(deps): update rust crate zip to v5 (#4551) @renovate[bot]
- Add more unchecked constructors (#4527) @connortsui20
- chore: enable vortex-compact in every benchmark (#4531) @danking
- chore(deps): update rust crate cc to v1.2.36 (#4538) @renovate[bot]
- Consolidate on futures crate (#4537) @gatesn
- Capitalize some words in the perf script (#4519) @AdamGS
- chore(deps): update rust crate wasm-bindgen-futures to v0.4.51 (#4533) @renovate[bot]
- chore(deps): update rust crate insta to v1.43.2 (#4532) @renovate[bot]
- Rename bug_report.md to bug_report.yml (#4526) @robert3005
- chore[ci]: natural change benchmark github post (#4518) @joseph-isaacs
- chore[ci]: make perf script less wordy (#4517) @joseph-isaacs
- chore(deps): update actions/setup-python action to v6 (#4515) @renovate[bot]
- fix(deps): update all patch updates to v1.0.175 (patch) (#4514) @renovate[bot]
- chore(deps): update aws-actions/configure-aws-credentials action to v5 (#4505) @renovate[bot]
- Allow bench results to collapse in GH comment (#4509) @gatesn
- chore(deps): update rust crate log to v0.4.28 (#4510) @renovate[bot]
- chore[ci]: tpch-10 on ci action (#4498) @joseph-isaacs
- chore[mask]: move mask future into vortex-array (#4496) @joseph-isaacs