fix: Remove fallback for maps containing complex types #2943
Conversation
| val partitionSchemaSupported = | ||
| typeChecker.isSchemaSupported(partitionSchema, fallbackReasons) | ||
|
|
||
| def hasUnsupportedType(dataType: DataType): Boolean = { |
There was a problem hiding this comment.
this check is no longer needed
| case t: StructType => t.exists(f => hasTemporalType(f.dataType)) | ||
| case t: ArrayType => hasTemporalType(t.elementType) |
There was a problem hiding this comment.
this was missing a check for MapType so I added that and moved the method to DataTypeSupport
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2943 +/- ##
============================================
+ Coverage 56.12% 59.63% +3.50%
- Complexity 976 1375 +399
============================================
Files 119 167 +48
Lines 11743 15487 +3744
Branches 2251 2566 +315
============================================
+ Hits 6591 9235 +2644
- Misses 4012 4955 +943
- Partials 1140 1297 +157 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| } | ||
|
|
||
| def hasTemporalType(t: DataType): Boolean = t match { | ||
| case DataTypes.DateType | DataTypes.TimestampType | DataTypes.TimestampNTZType => |
There was a problem hiding this comment.
wondering if interval can also be considered as temporal? Probably not, it is more like duration rather than representing any point of time
| var filename: String = null | ||
|
|
||
| /** Filename for data file with deeply nested complex types */ | ||
| var complexTypesFilename: String = null |
There was a problem hiding this comment.
would be great to add if this filename is input or output or temp location?
There was a problem hiding this comment.
I will improve the comments in my next PR later today
|
Thanks for the review @comphead |
3 participants
Which issue does this PR close?
N/A
Rationale for this change
Ongoing refactoring / fixing technical debt
What changes are included in this PR?
CometFuzzTestBasenow generates an additional Parquet file containing deeply nested typesCometScanRulehandling ofautomodeMapTypeHow are these changes tested?