Skip to content

Undeprecate ArrowWriter::into_serialized_writer and add docs #8621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 16, 2025
Merged

Undeprecate ArrowWriter::into_serialized_writer and add docs #8621

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c4f0fc1
Undeprecate ArrowWriter::into_serialized_writer and add docs
alamb Oct 15, 2025
619210d
Improve comments
alamb Oct 16, 2025
cdfe382
Merge remote-tracking branch 'apache/main' into alamb/into_serialized…
alamb Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions parquet/src/arrow/arrow_writer/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,11 +450,11 @@ impl<W: Write + Send> ArrowWriter<W> {
}

/// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].
/// This can be useful to provide more control over how files are written.
#[deprecated(
since = "57.0.0",
note = "Construct a `SerializedFileWriter` and `ArrowRowGroupWriterFactory` directly instead"
)]
///
/// Flushes any outstanding data before returning.
///
/// This can be useful to provide more control over how files are written, for example
/// to write columns in parallel. See the example on [`ArrowColumnWriter`].
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to drop a few breadcrumbs to find @adamreeve 's new example in #8582

pub fn into_serialized_writer(
mut self,
) -> Result<(SerializedFileWriter<W>, ArrowRowGroupWriterFactory)> {
Expand Down Expand Up @@ -872,6 +872,11 @@ impl ArrowColumnWriter {
}

/// Encodes [`RecordBatch`] to a parquet row group
///
/// You can create this structure via an [`ArrowRowGroupWriterFactory`]
///
/// See the example on [`ArrowColumnWriter`] for how to encode columns in parallel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ArrowRowGroupWriter is actually not public and not used in the ArrowColumnWriter example, so this last sentence might be a little confusing.

(ArrowRowGroupWriterFactory is internally used to create ArrowRowGroupWriters, but publicly it only exposes the ability to create a Vec<ArrowColumnWriter>. If I was starting this again from scratch I might have named things a little differently...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks -- I tried to improve the comments in 619210d

#[derive(Debug)]
struct ArrowRowGroupWriter {
writers: Vec<ArrowColumnWriter>,
schema: SchemaRef,
Expand Down Expand Up @@ -907,6 +912,10 @@ impl ArrowRowGroupWriter {
}

/// Factory that creates new column writers for each row group in the Parquet file.
///
/// You can create this structure via an [`ArrowWriter::into_serialized_writer`].
/// See the example on [`ArrowColumnWriter`] for how to encode columns in parallel
#[derive(Debug)]
pub struct ArrowRowGroupWriterFactory {
schema: SchemaDescPtr,
arrow_schema: SchemaRef,
Expand Down
Loading