Skip to content
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion parquet/src/arrow/array_reader/builder.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,13 +152,17 @@ impl<'a> ArrayReaderBuilder<'a> {
return Ok(Some(reader));
};

if cache_options.projection_mask.leaf_included(col_idx) {
// Skip caching for nested fields (inside List/Map) where rep_level > 0
// because the cache tracks by row count, but nested fields have
// values that don't correspond 1:1 with rows due to repetition
if cache_options.projection_mask.leaf_included(col_idx) && field.rep_level == 0 {
Ok(Some(Box::new(CachedArrayReader::new(
reader,
Arc::clone(cache_options.cache),
col_idx,
cache_options.role,
self.metrics.clone(), // cheap clone
field.def_level > 0, // needs_def_levels: true if has nullable ancestors
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively could pass field: &ParquetField or def_level: i16 and have CachedArrayReader::new figure out how to use that

))))
} else {
Ok(Some(reader))
Expand Down
Loading
Loading