v0.7.3

[desmondcheongzx]

What's Changed 🚀

✨ Features

• feat(observability): Support exporting Flotilla metrics @srilman (feat(observability): Support exporting Flotilla metrics #6122)
• feat: Nightly installations under nightly.daft.ai @desmondcheongzx (feat: Nightly installations under nightly.daft.ai #6175)
• feat: Support additional OTEL_* configuration envs @srilman (feat: Support additional OTEL_* configuration envs #6148)
• feat(lance): Support lance namespace read and write @shaofengshi (feat(lance): Support lance namespace read and write #5980)
• feat: add unity oauth m2m access token support @cckellogg (feat: add unity oauth m2m access token support #5839)
• feat: Support snapshot properties for Iceberg writes @desmondcheongzx (feat: Support snapshot properties for Iceberg writes #6139)
• feat: delimiter for agg_concat @aaron-ang (feat: delimiter for agg_concat #6099)
• feat: add uuid function @everySympathy (feat: add uuid function #5983)
• feat: add custom date and timestamp formatting for CSV writes @madvart (feat: add custom date and timestamp formatting for CSV writes #6073)
• feat(observability): Export metrics out as a table with the result @srilman (feat(observability): Export metrics out as a table with the result #6055)
• feat: comparison ops for list and struct types @aaron-ang (feat: comparison ops for list and struct types #6104)
• feat(frontend): enhance dashboard UI and fix Ray runner state reporting @Jay-ju (feat(frontend): enhance dashboard UI and fix Ray runner state reporting #6063)
• feat(dashboard): backend implementation @Jay-ju (feat(dashboard): backend implementation #6062)
• feat: string casing functions @aaron-ang (feat: string casing functions #6096)
• feat: list_contains expression @aaron-ang (feat: list_contains expression #6095)
• feat: specify text embedding dim @aaron-ang (feat: specify text embedding dim #6097)
• feat: Expr.var with ddof @aaron-ang (feat: Expr.var with ddof #6105)
• feat(lance): add nearest vector search support @huleilei (feat(lance): add nearest vector search support #6025)
• feat: distance and similarity functions @aaron-ang (feat: distance and similarity functions #6098)
• feat: add gravitino connector optional dependency @shaofengshi (feat: add gravitino connector optional dependency #6083)
• feat: Add support for ignoring null fields when writing json @gpathak128 (feat: Add support for ignoring null fields when writing json #6049)

🐛 Bug Fixes

• fix(docs): Fix broken links in modalities documentation @everettVT (fix(docs): Fix broken links in modalities documentation #6197)
• fix: Render Map columns as python dicts instead of list[struct] in terminal prints @srilman (fix: Render Map columns as python dicts instead of list[struct] in terminal prints #6198)
• fix: Fix merge conflict with OTEL configuration PR @srilman (fix: Fix merge conflict with OTEL configuration PR #6187)
• fix(observability): Truncate progress bar names by characters, not bytes @desmondcheongzx (fix(observability): Truncate progress bar names by characters, not bytes #6180)
• fix: fix concat.rs @colin-ho (fix: fix concat.rs #6174)
• fix(test): bump actor UDF timeout from 10s to 60s to reduce flakiness @ykdojo (fix(test): bump actor UDF timeout from 10s to 60s to reduce flakiness #6163)
• fix(optimizer): Fix bug with pushing filters through anti-joinsI s @desmondcheongzx (fix(optimizer): Fix bug with pushing filters through anti-joinsI s #6150)
• fix: Allow is_in to accept sets, tuples, and other iterables @desmondcheongzx (fix: Allow is_in to accept sets, tuples, and other iterables #6115)
• fix(ci): restore Rust code coverage by pinning cargo-llvm-cov @desmondcheongzx (fix(ci): restore Rust code coverage by pinning cargo-llvm-cov #6146)
• fix: make df.into_partitions() work when input num == num_partitions @everySympathy (fix: make df.into_partitions() work when input num == num_partitions #6061)
• fix(udf): ensure per-call kwargs in udf v2 are uniquely bound per call site @huleilei (fix(udf): ensure per-call kwargs in udf v2 are uniquely bound per call site #6079)
• fix(rustfmt): ignore parquet directories @aaron-ang (fix(rustfmt): ignore parquet directories #6101)
• fix: No need extra 1 for OffsetBufferBuilder @colin-ho (fix: No need extra 1 for OffsetBufferBuilder #6057)
• fix: use unique bucket names for running tests in parallel @rchowell (fix: use unique bucket names for running tests in parallel #6052)

🚀 Performance

• perf: zero copy on from_vec @universalmind303 (perf: zero copy on from_vec #6172)
• perf: Only Serialize Required Cols in Actor UDFs @plotor (perf: Only Serialize Required Cols in Actor UDFs #5884)

♻️ Refactor

• refactor(arrow2): migrate time.rs temporal methods to arrow-rs @rohitkulshreshtha (refactor(arrow2): migrate time.rs temporal methods to arrow-rs #6160)
• refactor(arrow2): migrate null checks to arrow-rs @rohitkulshreshtha (refactor(arrow2): migrate null checks to arrow-rs #6152)
• refactor(arrow2): migrates sparse-tensor to arrow-rs @cckellogg (refactor(arrow2): migrates sparse-tensor to arrow-rs #6179)
• refactor(swordfish): Separate concat and make streaming sink single input @colin-ho (refactor(swordfish): Separate concat and make streaming sink single input #6059)
• refactor(arrow2): migrates hll-sketch to arrow-rs @cckellogg (refactor(arrow2): migrates hll-sketch to arrow-rs #6169)
• refactor(arrow2): use arrow-rs for python conversions @universalmind303 (refactor(arrow2): use arrow-rs for python conversions #6130)
• refactor: Migrate hll_merge.rs from arrow2 to arrow-rs @rohitkulshreshtha (refactor: Migrate hll_merge.rs from arrow2 to arrow-rs #6158)
• refactor(arrow2): remove arrow2 from minhash @cckellogg (refactor(arrow2): remove arrow2 from minhash #6149)
• refactor: Migrate product.rs from arrow2 to arrow-rs @rohitkulshreshtha (refactor: Migrate product.rs from arrow2 to arrow-rs #6159)
• refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs @rchowell (refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs #6056)
• refactor(arrow2): migrate the extra easy kernels to arrow-rs @universalmind303 (refactor(arrow2): migrate the extra easy kernels to arrow-rs #6145)
• refactor(arrow2): migrate BinaryArray iceberg_truncate to arrow-rs @rohitkulshreshtha (refactor(arrow2): migrate BinaryArray iceberg_truncate to arrow-rs #6157)
• refactor(arrow2): migrate DataArray full_null/empty to arrow-rs @rohitkulshreshtha (refactor(arrow2): migrate DataArray full_null/empty to arrow-rs #6151)
• refactor(arrow2): remove a bunch of from impls that used arrow2 @universalmind303 (refactor(arrow2): remove a bunch of from impls that used arrow2 #6137)
• refactor(scalarudf): followup to uuid pr @universalmind303 (refactor(scalarudf): followup to uuid pr #6129)
• refactor(arrow-rs): Remove arrow2 from the search_sorted kernel @desmondcheongzx (refactor(arrow-rs): Remove arrow2 from the search_sorted kernel #6034)
• refactor(arrow-rs): Migrate approx count distinct @desmondcheongzx (refactor(arrow-rs): Migrate approx count distinct #6038)
• refactor(arrow-rs): migrate is_in and get_lit ops to Arrow-rs arrays @huleilei (refactor(arrow-rs): migrate is_in and get_lit ops to Arrow-rs arrays #6085)
• refactor: Migrate utf8 left function to arrow-rs @huleilei (refactor: Migrate utf8 left function to arrow-rs #6004)
• refactor(arrow2): migrate daft-functions-utf8/split to arrow-rs @colin-ho (refactor(arrow2): migrate daft-functions-utf8/split to arrow-rs #6046)
• refactor(arrow-rs): Migrate sketch_percentile kernel @desmondcheongzx (refactor(arrow-rs): Migrate sketch_percentile kernel #6044)
• refactor(swordfish): Separate joins @colin-ho (refactor(swordfish): Separate joins #6042)
• refactor(arrow2): remove some arrow2 based from impls @universalmind303 (refactor(arrow2): remove some arrow2 based from impls #6054)
• refactor(arrow2): migrate sum, min, max agg kernels @kevinzwang (refactor(arrow2): migrate sum, min, max agg kernels #6045)
• refactor(arrow2): utf8 comparison kernels @kevinzwang (refactor(arrow2): utf8 comparison kernels #6032)
• refactor(flotilla): Use stream::iter instead of channel in flotilla source nodes @colin-ho (refactor(flotilla): Use stream::iter instead of channel in flotilla source nodes #6043)
• refactor(arrow2): remove arrow2 *_array() methods from ImageArray @universalmind303 (refactor(arrow2): remove arrow2 *_array() methods from ImageArray #6050)
• refactor(arrow-rs): Migrate binary from_iter methods @desmondcheongzx (refactor(arrow-rs): Migrate binary from_iter methods #6040)
• refactor(observability): Refactor StatSnapshot to be predefined structs @srilman (refactor(observability): Refactor StatSnapshot to be predefined structs #6033)

📖 Documentation

• docs: Add daft.File usage throughout modalities @everettVT (docs: Add daft.File usage throughout modalities #6074)
• docs: Governance proposed changes @jaychia (docs: Governance proposed changes #6117)
• docs: Clarify imagenet benchmark setup @desmondcheongzx (docs: Clarify imagenet benchmark setup #6147)
• docs: add TosConfig documentation reference in config.md @huleilei (docs: add TosConfig documentation reference in config.md #6068)
• docs: Update daft.File API and add docstrings, improve error handling in tests @everettVT (docs: Update daft.File API and add docstrings, improve error handling in tests #5877)
• docs: add mm structured outputs tutorial @everettVT (docs: add mm structured outputs tutorial #5816)
• docs: Add end-to-end image pipeline example and regression test @huleilei (docs: Add end-to-end image pipeline example and regression test #6006)

✅ Tests

• test: Add missing result parameter to on_query_end in Google AI test @desmondcheongzx (test: Add missing result parameter to on_query_end in Google AI test #6184)

👷 CI

• ci: Exclude common-arrow-ffi from rust tests @desmondcheongzx (ci: Exclude common-arrow-ffi from rust tests #6173)
• ci: Accept pandas StringDtype in schema override tests @desmondcheongzx (ci: Accept pandas StringDtype in schema override tests #6165)
• ci: Add pyarrow to wheel build test dependencies @desmondcheongzx (ci: Add pyarrow to wheel build test dependencies #6109)
• ci: Add pytz and numpy to wheel build test dependencies @desmondcheongzx (ci: Add pytz and numpy to wheel build test dependencies #6108)
• ci: Fix nightly workflow permissions @desmondcheongzx (ci: Fix nightly workflow permissions #6037)

🔧 Maintenance

• chore: Intermediate op single input @colin-ho (chore: Intermediate op single input #6189)
• chore: Revert "refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs" @universalmind303 (chore: Revert "refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs" #6164)
• chore: Remove ci/ folder @srilman (chore: Remove ci/ folder #6154)
• chore: agg_concat kernel nits @colin-ho (chore: agg_concat kernel nits #6136)
• chore: Remove vendored parquet-format-safe, use upstream patched version @desmondcheongzx (chore: Remove vendored parquet-format-safe, use upstream patched version #6118)
• chore(deps): bump the all group with 5 updates @dependabot[bot] (chore(deps): bump the all group with 5 updates #5908)
• chore: Allows make clean to skip cleaning the python virtual environment @plotor (chore: Allows make clean to skip cleaning the python virtual environment #6103)
• chore: correct comment of enable_scan_task_split_and_merge in set_execution_config @everySympathy (chore: correct comment of enable_scan_task_split_and_merge in set_execution_config #6077)
• chore: Update lru dependency @desmondcheongzx (chore: Update lru dependency #6041)

⬆️ Dependencies

• chore(deps): bump the all group with 5 updates @dependabot[bot] (chore(deps): bump the all group with 5 updates #5908)

Full Changelog: v0.7.2...v0.7.3

---

This discussion was created from the release v0.7.3.