DataFusion 52 migration #3052
DataFusion 52 migration #3052
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3052 +/- ##
============================================
- Coverage 56.12% 9.97% -46.16%
+ Complexity 976 234 -742
============================================
Files 119 167 +48
Lines 11743 15508 +3765
Branches 2251 2549 +298
============================================
- Hits 6591 1547 -5044
- Misses 4012 13784 +9772
+ Partials 1140 177 -963 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| file_source, | ||
| ) | ||
| .with_projection_indices(Some(projection_vector)) | ||
| .with_table_partition_cols(partition_fields) |
There was a problem hiding this comment.
with_partition_cols now derived automatically
https://datafusion.apache.org/library-user-guide/upgrading.html#refactoring-of-filesource-constructors-and-filescanconfigbuilder-to-accept-schemas-upfront
| make_decimal_scalar(a, precision, scale, &f) | ||
| } | ||
| ScalarValue::Float32(_) | ScalarValue::Float64(_) => Ok(ColumnarValue::Scalar( | ||
| ScalarValue::try_from_array(&round(&[a.to_array()?, args[1].to_array(1)?])?, 0)?, |
There was a problem hiding this comment.
the direct function is private now, creating UDF instead
| } | ||
|
|
||
| let source = ParquetSource::default().with_schema_adapter_factory(Arc::new( | ||
| SparkSchemaAdapterFactory::new( |
There was a problem hiding this comment.
If you're removing a SparkSchemaAdapterFactory you need to replace it with a SparkPhysicalExprAdapterFactory.
There was a problem hiding this comment.
its moved to 4015 line on FileScanConfigBuilder level
| // Determine the schema to use for ParquetSource | ||
| let table_schema = if let Some(ref data_schema) = data_schema { |
There was a problem hiding this comment.
There is also TableSchema::with_table_partition_cols which might make this easier
3 participants
Which issue does this PR close?
Closes #3046 .
Rationale for this change
What changes are included in this PR?
How are these changes tested?