Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,14 @@ rust-version = "1.88"
anyhow = "1.0.72"
apache-avro = { version = "0.21", features = ["zstandard"] }
array-init = "2"
arrow-arith = "57.0"
arrow-array = "57.0"
arrow-buffer = "57.0"
arrow-cast = "57.0"
arrow-ord = "57.0"
arrow-schema = "57.0"
arrow-select = "57.0"
arrow-string = "57.0"
arrow-arith = "57.1"
arrow-array = "57.1"
arrow-buffer = "57.1"
arrow-cast = "57.1"
arrow-ord = "57.1"
arrow-schema = "57.1"
arrow-select = "57.1"
arrow-string = "57.1"
Comment on lines +45 to +52

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the associated linked issue: #1934
I see @lgingerich mentioned
so there is no crate dependency upgrade work that needs done

do we still need this pckage upgrade to be done as a part of this change

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the upgrade to 57.1 is required. The ArrowReaderOptions::metadata_options() method used in this PR was only exposed starting in arrow 57.1 (see apache/arrow-rs#7393). Without the upgrade, the code won't compile.

as-any = "0.3.2"
async-trait = "0.1.89"
aws-config = "1.8.7"
Expand Down Expand Up @@ -101,7 +101,7 @@ num-bigint = "0.4.6"
once_cell = "1.20"
opendal = "0.55.0"
ordered-float = "4"
parquet = "57.0"
parquet = "57.1"
pilota = "0.11.10"
port_scanner = "0.1.5"
pretty_assertions = "1.4"
Expand Down
49 changes: 25 additions & 24 deletions bindings/python/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 6 additions & 4 deletions crates/iceberg/src/arrow/reader.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1705,19 +1705,21 @@ impl<R: FileRead> AsyncFileReader for ArrowFileReader<R> {
)
}

// TODO: currently we don't respect `ArrowReaderOptions` cause it don't expose any method to access the option field
// we will fix it after `v55.1.0` is released in https://github.com/apache/arrow-rs/issues/7393
fn get_metadata(
&mut self,
_options: Option<&'_ ArrowReaderOptions>,
options: Option<&'_ ArrowReaderOptions>,
) -> BoxFuture<'_, parquet::errors::Result<Arc<ParquetMetaData>>> {
// Extract metadata options from ArrowReaderOptions if provided
let metadata_options = options.map(|o| o.metadata_options().clone());

async move {
let reader = ParquetMetaDataReader::new()
.with_prefetch_hint(self.metadata_size_hint)
// Set the page policy first because it updates both column and offset policies.
.with_page_index_policy(PageIndexPolicy::from(self.preload_page_index))
.with_column_index_policy(PageIndexPolicy::from(self.preload_column_index))
.with_offset_index_policy(PageIndexPolicy::from(self.preload_offset_index));
.with_offset_index_policy(PageIndexPolicy::from(self.preload_offset_index))
.with_metadata_options(metadata_options);
let size = self.meta.size;
let meta = reader.load_and_finish(self, size).await?;

Expand Down
Loading