fix[array]: correct the handling of to_arrow execution and struct casting #5867
Conversation
…ting to expand nullable fields missing in the struct Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
CodSpeed Performance ReportMerging #5867 will degrade performance by 11.37%Comparing Summary
Benchmarks breakdown
Footnotes
|
Signed-off-by: Joe Isaacs <[email protected]>
Codecov Report❌ Patch coverage is ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| .map(move |chunk| { | ||
| if *USE_VORTEX_OPERATORS { | ||
| chunk.execute_record_batch(&projected_schema, &chunk_session) | ||
| let schema = chunk.dtype().to_arrow_schema()?; |
There was a problem hiding this comment.
you can get the schema once from the scan and projection expression and use it here (I'm doing it in the DF52 PR, so not a blocker)
| ) -> VortexResult<ArrowArrayRef> { | ||
| let mut arrow_fields = Vec::with_capacity(vortex_fields.len()); | ||
| for (field, vx_field) in fields.iter().zip(vortex_fields.into_iter()) { | ||
| for (field, vx_field) in fields.iter().zip_eq(vortex_fields.into_iter()) { |
| ParentRuleSet::new(&[ParentRuleSet::lift(&ScalarFnUnaryFilterPushDownRule)]); | ||
|
|
||
| /// Converts a ScalarFnArray with Pack into a StructArray directly. | ||
| #[derive(Debug)] |
There was a problem hiding this comment.
Why is derive(debug) needed?
Expand nullable fields missing in the struct