Skip to content

Commit ba21a83

Browse files
authored
[Parquet] Optimize appending max level comparison in DefinitionLevelDecoder (#9217)
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #9216. # Rationale for this change Profiling showed this loop to be a clear performance hotspot and thanks to the new `BooleanBufferBuilder::extend_trusted_len` method introduced in #9137 there is a very simple improvement. Benchmark results for optional structs after the fix: ``` arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, no NULLs time: [69.873 µs 69.917 µs 69.970 µs] change: [−60.075% −60.046% −60.018%] (p = 0.00 < 0.05) Performance has improved. arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, half NULLs time: [136.62 µs 136.66 µs 136.72 µs] change: [−67.663% −67.536% −67.416%] (p = 0.00 < 0.05) Performance has improved. ``` This is a big improvement, but still significantly slower than reading non-nested data. The main hotspot still is the `extend_trusted_len` and manual simd code for comparing chunks and appending 64-bits at a time could potentially speed it up even more. But that would require either architecture specific intrinsics or unstable features. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? Tested by existing tests. # Are there any user-facing changes? no
1 parent ba3446b commit ba21a83

File tree

2 files changed

+11
-5
lines changed

2 files changed

+11
-5
lines changed

parquet/src/arrow/array_reader/struct_array.rs

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -158,8 +158,13 @@ impl ArrayReader for StructArrayReader {
158158
}
159159
}
160160
None => {
161-
for def_level in def_levels {
162-
bitmap_builder.append(*def_level >= self.struct_def_level)
161+
// Safety: slice iterator has a trusted length
162+
unsafe {
163+
bitmap_builder.extend_trusted_len(
164+
def_levels
165+
.iter()
166+
.map(|level| *level >= self.struct_def_level),
167+
)
163168
}
164169
}
165170
}

parquet/src/arrow/record_reader/definition_levels.rs

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -160,9 +160,10 @@ impl DefinitionLevelDecoder for DefinitionLevelBufferDecoder {
160160
let start = levels.len();
161161
let (values_read, levels_read) = decoder.read_def_levels(levels, num_levels)?;
162162

163-
nulls.reserve(levels_read);
164-
for i in &levels[start..] {
165-
nulls.append(i == max_level);
163+
// Safety: slice iterator has a trusted length
164+
unsafe {
165+
nulls
166+
.extend_trusted_len(levels[start..].iter().map(|level| level == max_level));
166167
}
167168

168169
Ok((values_read, levels_read))

0 commit comments

Comments
 (0)