Skip to content

Conversation

@Li0k
Copy link

@Li0k Li0k commented Nov 27, 2025

This pull request introduces several improvements and refactorings to the Iceberg writer modules, focusing on making the codebase more modular and robust. The most significant changes include safer handling of optional writer internals, the addition of new functional and task-based writer modules, and updates to dependencies to support new features.

Writer robustness and refactoring

  • Refactored current_row_num and current_written_size methods in data_file_writer.rs, position_delete_file_writer.rs, and rolling_writer.rs to use safer handling of optional internals (Option) by replacing unwrap() with map_or, preventing potential panics when the inner writer is absent. [1] [2] [3]

Modularization and new writer modules

  • Added new modules: function_writer (with equality_delta_writer) and task_writer to crates/iceberg/src/writer/mod.rs, making the writer architecture more extensible and organized. [1] [2] [3]
  • Updated visibility of the parquet_writer module to pub in file_writer/mod.rs, allowing external usage and extension.

Dependency updates

  • Added arrow-row = "56.2" to both the workspace and crate-specific Cargo.toml files to support new features in Arrow-based row handling. [1] [2]

Cleanup

  • Removed the now-unnecessary task_writer module from crates/integrations/datafusion/src/lib.rs following its migration to the core writer module.## Which issue does this PR close?

@Li0k Li0k requested a review from chenzl25 November 27, 2025 09:59
@chenzl25 chenzl25 changed the title feat: expose task writer feat: expose task writer & support equality delta writer Dec 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants