v0.7.6 · Eventual-Inc Daft · Discussion #6538 · GitHub

rchowell
Mar 30, 2026
Maintainer

What's Changed 🚀

💥 Breaking Changes

feat!: add iceberg table support with the gravitino catalog @rchowell (feat!: add iceberg table support with the gravitino catalog #6509)

✨ Features

feat!: add iceberg table support with the gravitino catalog @rchowell (feat!: add iceberg table support with the gravitino catalog #6509)
feat(swordfish): plan caching @colin-ho (feat(swordfish): plan caching #6278)
feat: add daft-checkpoint crate with CheckpointStore trait and in-mem… @rohitkulshreshtha (feat: add daft-checkpoint crate with CheckpointStore trait and in-mem… #6511)
feat: optionally keep the native progress bar visible after job completion @BABTUNA (feat: optionally keep the native progress bar visible after job completion #6449)
feat(build): add experimental cross-wheel build target @cckellogg (feat(build): add experimental cross-wheel build target #6360)
feat: DataFrame.from_files @rchowell (feat: DataFrame.from_files #6507)
feat: Add Apache Paimon Lake Format Support @chenghuichen (feat: Add Apache Paimon Lake Format Support #6450)
feat: add current_date, current_timestamp, current_timezone and SQL aliases @BABTUNA (feat: add current_date, current_timestamp, current_timezone and SQL aliases #6495)
feat(subscribers): add typed event for operator and stats callbacks @cckellogg (feat(subscribers): add typed event for operator and stats callbacks #6479)
feat: support to_ray_dataset() from the native runner @desmondcheongzx (feat: support to_ray_dataset() from the native runner #6486)
feat: async data sources and tasks @rchowell (feat: async data sources and tasks #6492)
feat: Add hex and unhex @Lucas61000 (feat: Add hex and unhex #6373)
feat: implement _repr_mimebundle_ for DataFrame @Abyss-lord (feat: implement _repr_mimebundle_ for DataFrame #6385)
feat: Add string strip function @Lucas61000 (feat: Add string strip function #6372)
feat: add plan fingerprinting for LocalPhysicalPlan @colin-ho (feat: add plan fingerprinting for LocalPhysicalPlan #6276)
feat: Add metrics for join operators @srilman (feat: Add metrics for join operators #6391)
feat(observability): add process-level memory and CPU monitoring @desmondcheongzx (feat(observability): add process-level memory and CPU monitoring #6428)
feat: define a single DataSource trait matching our python interface. @rchowell (feat: define a single DataSource trait matching our python interface. #6427)
feat(io): Propagate storage_options to Lance backend to enable Apache Gravitino (gvfs://) support @qingfeng-occ (feat(io): Propagate storage_options to Lance backend to enable Apache Gravitino (gvfs://) support #6430)
feat(subscribers): add JSONL event log subscriber @cckellogg (feat(subscribers): add JSONL event log subscriber #6420)
feat: Add support for reading whole text files to read_text @plotor (feat: Add support for reading whole text files to read_text #6354)
feat(expressions): support count(mode='all') without expr @Abyss-lord (feat(expressions): support count(mode='all') without expr #6358)
feat: kafka bounded datasource @everySympathy (feat: kafka bounded datasource #5970)
feat(udf): support ray_options and resource overrides in UDF v2 @Jay-ju ( feat(udf): support ray_options and resource overrides in UDF v2 #5982)
feat: Add global agg support for Dataframe select @caican00 (feat: Add global agg support for Dataframe select #6067)

🐛 Bug Fixes

fix: correct grid card indentation on modalities overview page @everettVT (fix: correct grid card indentation on modalities overview page #6530)
fix(series): unable to cast directly from string to fixed_size_binary @Abyss-lord (fix(series): unable to cast directly from string to fixed_size_binary #6521)
fix: respect scheme aliases in Iceberg writes @rchowell (fix: respect scheme aliases in Iceberg writes #6512)
fix(series): cast should not change input argument name @Abyss-lord (fix(series): cast should not change input argument name #6515)
fix: preserve show options without setting format @aaron-ang (fix: preserve show options without setting format #6102)
fix: strict inequalities must NOT be relaxed for Identity transform @gavin9402 (fix: strict inequalities must NOT be relaxed for Identity transform #6484)
fix: fix column missing issue for jsonl record schema when reading json/jsonl files @caican00 (fix: fix column missing issue for jsonl record schema when reading json/jsonl files #5681)
fix: remove use_safetensors=True to allow model fallback @rchowell (fix: remove use_safetensors=True to allow model fallback #6489)
fix: close connections in io tests and address other warnings @rchowell (fix: close connections in io tests and address other warnings #6470)
fix: deflake a probabilistic test that is causing release uploads to fail @rchowell (fix: deflake a probabilistic test that is causing release uploads to fail #6475)
fix(io): fix JSONL byte-range reading and make chunk_size byte-based @everySympathy (fix(io): fix JSONL byte-range reading and make chunk_size byte-based #6374)
fix(sql): fall back to single scan when partition bounds are degenerate @qingfeng-occ (fix(sql): fall back to single scan when partition bounds are degenerate #6482)
fix: clippy errors on iterators in tests @rchowell (fix: clippy errors on iterators in tests #6476)
fix: handles schema inference errors when partition columns are NULL @rchowell (fix: handles schema inference errors when partition columns are NULL #6467)
fix: keep field metadata when converting series to ffi schema @universalmind303 (fix: keep field metadata when converting series to ffi schema #6471)
fix: Track durations for all local operators in distributed @srilman (fix: Track durations for all local operators in distributed #6438)
fix(optimizer): allow column-alias projections in join reordering @desmondcheongzx (fix(optimizer): allow column-alias projections in join reordering #6412)
fix: Expose json_target_filesize execution config @plotor (fix: Expose json_target_filesize execution config #6382)
fix: fix for bad merge @universalmind303 (fix: fix for bad merge #6419)
fix(sql): Fix error in TPC-DS q23 @Lucas61000 (fix(sql): Fix error in TPC-DS q23 #6317)
fix: Preserve materialized stats when splitting granular projections @plotor (fix: Preserve materialized stats when splitting granular projections #6376)
fix(dataframe): preserve field order in from_pylist @Abyss-lord (fix(dataframe): preserve field order in from_pylist #6369)
fix: report bytes.read for local CSV scans @Abyss-lord (fix: report bytes.read for local CSV scans #6380)

🚀 Performance

perf: Optimize list_* aggregation functions @srilman (perf: Optimize list_* aggregation functions #6494)
perf: introduce Column type with ScalarColumn for O(1) scalar storage in RecordBatch @universalmind303 (perf: introduce Column type with ScalarColumn for O(1) scalar storage in RecordBatch #6444)

♻️ Refactor

refactor: implement generic shuffle interface for flight shuffle @ohbh (refactor: implement generic shuffle interface for flight shuffle #6456)
refactor(ext): use single, publishable crate for daft-ext types @rchowell (refactor(ext): use single, publishable crate for daft-ext types #6452)
refactor(optimizer): introduce bitmask-based RelationSet for join reordering @desmondcheongzx (refactor(optimizer): introduce bitmask-based RelationSet for join reordering #6421)
refactor(io): merge conditional arms in opendal_backends ls operator @qingfeng-occ (refactor(io): merge conditional arms in opendal_backends ls operator #6461)
refactor(io): move common source fields to own type @rchowell (refactor(io): move common source fields to own type #6451)
refactor: remove EmptyScan plan node @colin-ho (refactor: remove EmptyScan plan node #6275)
refactor(swordfish): Streaming flight shuffle read @colin-ho (refactor(swordfish): Streaming flight shuffle read #6269)

📖 Documentation

docs: clean up roadmap wording @ykdojo (docs: clean up roadmap wording #6517)
docs: update roadmap for 2026 @everettVT (docs: update roadmap for 2026 #6480)
docs: fix daft dashboard command example @qingfeng-occ (docs: fix daft dashboard command example #6483)
docs: document get_or_infer_runner_type and get_or_create_runner @ykdojo (docs: document get_or_infer_runner_type and get_or_create_runner #6453)
docs(kafka): add dedicated connector docs for kafka @everySympathy (docs(kafka): add dedicated connector docs for kafka #6433)
docs: improve skills install instructions @ykdojo (docs: improve skills install instructions #6422)
docs: add Daft Skills page and Claude Code plugin config @ykdojo (docs: add Daft Skills page and Claude Code plugin config #6368)

👷 CI

ci: disable HF xet storage backend in unit tests @rchowell (ci: disable HF xet storage backend in unit tests #6493)
ci: refresh apt index before installing deps @rchowell (ci: refresh apt index before installing deps #6491)
ci: remove broken links @rchowell (ci: remove broken links #6490)

🔧 Maintenance

chore: exclude symlink files from end-of-file-fixer pre-commit hook @chenghuichen (chore: exclude symlink files from end-of-file-fixer pre-commit hook #6531)
chore(ci): migrate BuildJet runners to Blacksmith @madvart (chore(ci): migrate BuildJet runners to Blacksmith #6416)
chore: use dedicated OSS AWS account @jeevb (chore: use dedicated OSS AWS account #6442)
chore: Collection of random Swordfish cleanups @srilman (chore: Collection of random Swordfish cleanups #6454)
chore(swordfish): Small swordfish cleanups @colin-ho (chore(swordfish): Small swordfish cleanups #6488)
chore: use concurrent broken link checker @rchowell (chore: use concurrent broken link checker #6474)
chore: remove _FixEmptyStructArrays workaround @universalmind303 (chore: remove _FixEmptyStructArrays workaround #6478)
chore: deprecate all extra lance functions in favor of new daft-lance @universalmind303 (chore: deprecate all extra lance functions in favor of new daft-lance #6487)
chore: remove deprecated catalog modules @rchowell (chore: remove deprecated catalog modules #6457)
chore: Disable cargo check in local dev @srilman (chore: Disable cargo check in local dev #6439)
chore: Abstract metrics meters for execution @srilman (chore: Abstract metrics meters for execution #6384)
chore: Remove some extra & duplicate Rust packages @srilman (chore: Remove some extra & duplicate Rust packages #6435)
chore: Make the partition threshold to use pre_shuffle_merge configurable @plotor (chore: Make the partition threshold to use pre_shuffle_merge configurable #6356)
chore: rename DataSource enum to ScanSource @rchowell (chore: rename DataSource enum to ScanSource #6417)
chore: separates source types from file format configuration @rchowell (chore: separates source types from file format configuration #6406)

Full Changelog: v0.7.5...v0.7.6

This discussion was created from the release v0.7.6.

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment