Skip to content

Commit 1c487f7

Browse files
viiryacloud-fan
authored andcommitted
[SPARK-24762][SQL][FOLLOWUP] Enable Option of Product encoders
## What changes were proposed in this pull request? This is follow-up of apache#21732. This patch inlines `isOptionType` method. ## How was this patch tested? Existing tests. Closes apache#23143 from viirya/SPARK-24762-followup. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent c995e07 commit 1c487f7

File tree

1 file changed

+3
-6
lines changed

1 file changed

+3
-6
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -251,19 +251,16 @@ case class ExpressionEncoder[T](
251251
*/
252252
def isSerializedAsStruct: Boolean = objSerializer.dataType.isInstanceOf[StructType]
253253

254-
/**
255-
* Returns true if the type `T` is an `Option` type.
256-
*/
257-
def isOptionType: Boolean = classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass)
258-
259254
/**
260255
* If the type `T` is serialized as a struct, when it is encoded to a Spark SQL row, fields in
261256
* the struct are naturally mapped to top-level columns in a row. In other words, the serialized
262257
* struct is flattened to row. But in case of the `T` is also an `Option` type, it can't be
263258
* flattened to top-level row, because in Spark SQL top-level row can't be null. This method
264259
* returns true if `T` is serialized as struct and is not `Option` type.
265260
*/
266-
def isSerializedAsStructForTopLevel: Boolean = isSerializedAsStruct && !isOptionType
261+
def isSerializedAsStructForTopLevel: Boolean = {
262+
isSerializedAsStruct && !classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass)
263+
}
267264

268265
// serializer expressions are used to encode an object to a row, while the object is usually an
269266
// intermediate value produced inside an operator, not from the output of the child operator. This

0 commit comments

Comments
 (0)