🌪 0.42.0

Changes

⚠️ Breaks

• chore: Add new_expr to all built-in expressions in the name of consistency (#3847) @AdamGS

✨ Features

• feat[duckdb]: impl get_partition_data (#4008) @joseph-isaacs
• feat: teach SparseArray to canonicalize Binary arrays (#3993) @danking
• Add rayon parallel iterators for scanning (#3983) @gatesn
• feat[duckdb]: add file pruning and fix scanner (#3986) @joseph-isaacs
• Add a work-stealing ArrayIterator to drive scans (#3982) @gatesn
• feature: Add IsConstant for DecimalBytePartsArray (#3972) @robert3005
• feat: Add traversal logic for more containers (#3970) @AdamGS
• feat: Refactor traversal logic (#3926) @AdamGS
• DuckDB Table Filters (#3951) @gatesn
• feat: duckdb workstealing (#3927) @0ax1
• feat: use VarBinViewBuilder internally (#3925) @a10y
• Vortex - C++ API (#3792) @XinyuZeng
• feat: display as table (#3906) @blaginin
• propagate spans in dispatcher (#3916) @onursatici
• feat: duckdb global conversion cache (#3901) @0ax1
• feat: impl optimize for more arrays (#3909) @a10y
• feat: call optimize in compressor (#3852) @a10y
• feat: build with stable rust (#3881) @a10y
• feature: multi-file-scan thread-safe iterator (#3863) @0ax1
• Compact Compression (#3765) @mwlon
• Arrow iterator conversions (#3875) @gatesn
• feat: display (#3866) @danking
• feat[bench]: inter-query memory tracking (#3861) @joseph-isaacs
• feat: Minimal support for extension array casting (#3851) @AdamGS
• feat[compressor]: wire up sequence array into the compressor (#3844) @joseph-isaacs
• feat: teach utf8-type SparseArray to canonicalize (#3848) @danking
• feat: teach SparseArray to canonicalize Null & Decimal (#3662) @danking
• feat: Add optimize ArrayOp with VBView implementation (#3825) @a10y
• Row idx layout reader (#3819) @gatesn
• feature: Support overriding layout verification settings via environment variables (#3794) @robert3005
• Add expression transformation to replace all occurrences of a node (#3795) @gatesn
• VortexExpr VTables (#3713) @gatesn

🚀 Performance

• DuckDB Dynamic Expressions (#3966) @gatesn
• Add cardinality callback to DuckDB TableFunction (#3941) @gatesn
• perf: fine tune workstealing (#3938) @0ax1
• perf[vortex-duckdb]: set dictionary id to allow hash value once opt (#3621) @joseph-isaacs

🐛 Bug Fixes

35 changes

• fix: unify duckdb timestamps on i64 (#4004) @0ax1
• fix: Decimal256 from arrow casts the array to Decimal256Type (#4003) @robert3005
• fix: increment num_factories_constructed on error (#4001) @0ax1
• fix: Pruning expressions check NanCount where appropriate (#3973) @a10y
• fix: Make NotExpr display more explicit (#3998) @AdamGS
• fix[duckdb]: handle empty file globs (#3994) @joseph-isaacs
• fix: list array builder respects sliced list arrays (#3991) @danking
• feat[duckdb]: add file pruning and fix scanner (#3986) @joseph-isaacs
• fix: correctly expose gen_test_data cxx ffi (#3969) @XinyuZeng
• prevent double stealing (#3984) @onursatici
• fix: Prevent compressor from choosing SequenceArray for dictionary codes (#3979) @robert3005
• Fix DuckDB timestamp values (#3965) @gatesn
• Fix data_wrapper pointer (#3955) @gatesn
• fix: remove optimize Op, replace with compact_buffers function (#3931) @a10y
• fix: nbytes returns a u64 (#3924) @robert3005
• fix: duckdb multithreaded scan (#3920) @0ax1
• fix: duckdb scan cleanup (#3912) @0ax1
• fix: Fix NNF nested negation rewriting (#3905) @AdamGS
• fix: DictMaskEvaluation to respect validity (#3894) @blaginin
• fix: StructArray with empty fields is considered constant (#3895) @robert3005
• fix[layout]: row id layout reader cannot stop if inner layout registering its splits (#3867) @joseph-isaacs
• fix: Drain prefetch buffer (#3856) @a10y
• fix: teach SparseArray to account for null values when the fill is null (#3846) @danking
• fix[bench-website]: allow sf1000 and missing results (#3849) @joseph-isaacs
• fix[bench]: website render data with scale factor as string (#3838) @joseph-isaacs
• fix[bench]: correct s3 path to run benchmarks (#3837) @joseph-isaacs
• fix[bench]: nightly config that doesn't fail (#3830) @joseph-isaacs
• Fix empty take for ChunkedArray (#3823) @blaginin
• fix: Propagate executor to scan tasks (#3815) @AdamGS
• fix: Add test for using correct self._dataset variable for scanner schema (#3812) @NeejWeej
• fix: Remove redundant copy from delta compression (#3797) @robert3005
• fix: make Java test gradle self-contained (#3791) @a10y
• fix: DecimalByteParts compare with overflow and nullable elements (#3789) @a10y
• fix: STRIDE for u8/i32 AVX2 take kernel (#3771) @a10y
• fix: Correct Iterator::size_hint and ExactSizeIterator implementation for FieldNames (#3768) @AdamGS

🧰 Maintenance

110 changes

• docs: StructFields behavior (#4000) @a10y
• chore: Add some tests and docs to layouts and duckdb (#3999) @AdamGS
• chore[duckdb]: scan log info -> trace (#3989) @joseph-isaacs
• chore: RunEndArray ends must be unsigned so only enumerate unsigned ptypes for decompression (#3985) @robert3005
• chore: duckdb decimal to arrow conversion (#3980) @0ax1
• Extract work queue logic (#3978) @gatesn
• chore: check Struct Scalar dtype on creation (#3974) @blaginin
• chore: Convert cases of Buffer::extend to using extend_trusted (#3971) @robert3005
• chore(deps): update plugin com.diffplug.spotless to v7.2.1 (#3961) @renovate[bot]
• chore[bench]: assert partial taxi take result correctness (#3948) @joseph-isaacs
• chore: Implement FileSource::fmt_extra for VortexSource to display predicates in query plans (#3959) @robert3005
• Setup JDK in CI (#3967) @gatesn
• Use c-unwrap ABI for duckdb extension (#3956) @gatesn
• chore: All runs-on workflows have unique tags (#3957) @robert3005
• Improve work stealing loop (#3946) @gatesn
• chore(deps): update rust crate divan to v3.0.4 (#3950) @renovate[bot]
• chore: Propagate traces to task executor and a bunch of other places (#3947) @AdamGS
• chore(deps): update junit-framework monorepo to v5.13.4 (patch) (#3945) @renovate[bot]
• chore(deps): lock file maintenance (#3944) @renovate[bot]
• chore(deps): update rust crate rand to v0.9.2 (#3942) @renovate[bot]
• chore(deps): update plugin com.diffplug.spotless to v7.2.0 (#3943) @renovate[bot]
• chore(deps): update rust crate strum to v0.27.2 (#3939) @renovate[bot]
• fix(deps): update all patch updates to v1.0.161 (patch) (#3933) @renovate[bot]
• chore(deps): update rust crate serde_json to v1.0.141 (#3932) @renovate[bot]
• chore: fix type bound on Scalar::binary (#3930) @a10y
• chore: Fix c++ ci after it's a required check (#3928) @robert3005
• chore(deps): update all patch updates (patch) (#3910) @renovate[bot]
• chore: Enable workspace lints on vortex-scan (#3923) @robert3005
• Add BtrBlocksCompressor decision tree (#3921) @robert3005
• chore: Disable yamlint warnings (#3915) @AdamGS
• chore: Disable code coverage for arbitrary-related code (#3913) @AdamGS
• chore: Migrate NNF doctests to actual tests (#3911) @AdamGS
• chore: Fix fmt-failing duckdb code (#3914) @AdamGS
• chore: exclude arbitrary from coverage (#3908) @lwwmanning
• chore[docs]: deny missing_docs on vortex-scalar, vortex-utils, & vortex-flatbuffers (#3907) @lwwmanning
• feat[bench-vortex]: benchmark vortex compact format (#3886) @joseph-isaacs
• Use ptr::default in Rust 1.88 (#3896) @gatesn
• [chore] Update README.md (#3893) @lwwmanning
• chore[vortex-duckdb]: allow C++ 20 ext syntax (#3884) @joseph-isaacs
• duckdb build script http timeout (#3888) @onursatici
• chore[vortex-scan]: clippy default MultiScan (#3889) @joseph-isaacs
• chore(deps): update all patch updates (patch) (#3887) @renovate[bot]
• chore(config): migrate renovate config (#3883) @renovate[bot]
• chore: Don't generate compare expressions for lists when fuzzing (#3882) @robert3005
• fix(deps): update dependency org.immutables:value to v2.11.1 (#3880) @renovate[bot]
• chore: Consitent naming for display functions (#3871) @robert3005
• chore(deps): update rust crate fastlanes to 0.2.0 (#3878) @renovate[bot]
• chore[bench]: move tpcds into new benchmark format (#3870) @joseph-isaacs
• fix: Fuzzer does not generate filter on struct (#3873) @a10y
• chore(deps): update plugin com.palantir.consistent-versions to v2.36.0 (#3872) @renovate[bot]
• Fix intra-doc links (#3869) @gatesn
• chore[layout]: empty pack round trip scan (#3868) @joseph-isaacs
• chore: Replace dependency on top level arrow create with individual modules now that pyarrow is a standalone crate (#3864) @robert3005
• chore: Bump prost dependency (#3862) @AdamGS
• chore(deps): update plugin com.vanniktech.maven.publish to v0.34.0 (#3860) @renovate[bot]
• chore(deps): lock file maintenance (#3857) @renovate[bot]
• chore(deps): update rust crate memmap2 to v0.9.7 (#3855) @renovate[bot]
• Move array context inside FlatLayout (#3854) @gatesn
• chore: better naming for SendableLayoutFuture (#3843) @a10y
• chore: pull struct case of sparse to_canonical into function (#3850) @danking
• chore(deps): update rust crate divan to v3.0.3 (#3845) @renovate[bot]
• chore[fuzz]: smaller machine to save run cost m8g.2xlarge (#3842) @joseph-isaacs
• chore[ci]: add duckdb:duckdb to sf=10 run and sf=1000 run (#3840) @joseph-isaacs
• chore[ci]: remove clean up from CI (#3839) @joseph-isaacs
• chore: enable cxx compiler warnings in vortex-duckdb/build.rs (#3836) @0ax1
• chore: bump duckdb binaries to 1.3.2 (#3835) @0ax1
• chore(deps): update dependency duckdb/duckdb to v1.3.2 (#3833) @renovate[bot]
• DuckDB renovate (#3832) @gatesn
• chore: build time vendor duckdb (#3831) @0ax1
• chore[bench]: generate partitioned tpch files (#3824) @joseph-isaacs
• Remove var expression (#3829) @gatesn
• chore: TPC-H CI/coverage improvements (#3822) @lwwmanning
• chore[bench]: use scale factor as string (#3828) @joseph-isaacs
• chore: sanitize cxx build flags for vortex-duckdb (#3826) @0ax1
• chore: README improvements (#3820) @lwwmanning
• chore: add sf 0.01 & 0.1 and use 0.01 in ci (#3818) @joseph-isaacs
• Add precondition for VortexExpr::replace_children (#3817) @gatesn
• chore[fuzz]: run the fuzzer every 4 hours (#3816) @joseph-isaacs
• fix[bench]: update action and website (#3814) @joseph-isaacs
• chore[bench]: fix website and update scale factor options (#3813) @joseph-isaacs
• chore(deps): update rust crate clap to v4.5.41 (#3811) @renovate[bot]
• chore(deps): update rust crate zip to v4.3.0 (#3809) @renovate[bot]
• ci: update target dir for coverage runs (#3805) @a10y
• chore[bench]: nightly sql bench with different targets (sf=100) (#3808) @joseph-isaacs
• Generalize expression partition to annotationed expressions (#3803) @gatesn
• chore[bench]: parallel file gen (#3807) @joseph-isaacs
• chore[bench]: nightly run sf10 (#3806) @joseph-isaacs
• chore(deps): update plugin com.diffplug.spotless to v7.1.0 (#3790) @renovate[bot]
• chore[bench]: add tpchgen-rs benchmark generation (#3802) @joseph-isaacs
• chore: Remove TryIntoArray trait that was only used to convert RecordBatch into Vortex Array (#3804) @robert3005
• ci: run Rust tests with ASAN (#3801) @a10y
• chore[bench-vortex]: unify the different benchmark binaries (#3785) @joseph-isaacs
• chore: Remove clear-caches action (#3798) @robert3005
• ci: Fuzz failures should fail whole job (#3786) @a10y
• chore[bench]: store all run time measurements (#3784) @joseph-isaacs
• chore(deps): update datafusion and friends to v48.0.1 (patch) (#3783) @renovate[bot]
• chore(deps): lock file maintenance (#3780) @renovate[bot]
• chore(deps): update rust crate cc to v1.2.29 (#3778) @renovate[bot]
• Quality of life improvements (#3777) @gatesn
• chore(deps): update all patch updates (patch) (#3774) @renovate[bot]
• chore(deps): update fsfe/reuse-action action to v5 (#3773) @renovate[bot]
• chore: TakeKernel for List (#3750) @blaginin
• chore: SPDX license & copyrights (#3753) @lwwmanning
• chore(deps): update plugin com.palantir.consistent-versions to v2.35.0 (#3772) @renovate[bot]
• chore(deps): update rust crate cc to v1.2.28 (#3770) @renovate[bot]
• chore(deps): update junit-framework monorepo to v5.13.3 (patch) (#3769) @renovate[bot]
• chore: Tell fuzzer to use all cores on the machine (#3767) @robert3005
• chore[ci]: java ci is slow (#3758) @joseph-isaacs
• chore(deps): update rust crate indicatif to 0.18.0 (#3766) @renovate[bot]
• chore(deps): update rust crate libfuzzer-sys to v0.4.10 (#3763) @renovate[bot]