Skip to content

Commit 8f968b4

Browse files
Ngone51cloud-fan
authored andcommitted
[SPARK-26730][SQL] Strip redundant AssertNotNull for ExpressionEncoder's serializer
## What changes were proposed in this pull request? For types like Product, we've already add AssertNotNull when we construct serializer(see code below), so we could strip redundant AssertNotNull for those types. ``` val fieldValue = Invoke( AssertNotNull(inputObject, walkedTypePath), fieldName, dataTypeFor(fieldType), returnNullable = !fieldType.typeSymbol.asClass.isPrimitive) ``` ## How was this patch tested? Existed. Closes apache#23651 from Ngone51/dev-strip-redundant-assertnotnull-for-ecnoder-serializer. Authored-by: wuyi <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent cdd694c commit 8f968b4

File tree

1 file changed

+5
-2
lines changed

1 file changed

+5
-2
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -631,8 +631,11 @@ object ScalaReflection extends ScalaReflection {
631631
"cannot be used as field name\n" + walkedTypePath.mkString("\n"))
632632
}
633633

634-
val fieldValue = Invoke(
635-
AssertNotNull(inputObject, walkedTypePath), fieldName, dataTypeFor(fieldType),
634+
// SPARK-26730 inputObject won't be null with If's guard below. And KnownNotNul
635+
// is necessary here. Because for a nullable nested inputObject with struct data
636+
// type, e.g. StructType(IntegerType, StringType), it will return nullable=true
637+
// for IntegerType without KnownNotNull. And that's what we do not expect to.
638+
val fieldValue = Invoke(KnownNotNull(inputObject), fieldName, dataTypeFor(fieldType),
636639
returnNullable = !fieldType.typeSymbol.asClass.isPrimitive)
637640
val clsName = getClassNameFromType(fieldType)
638641
val newPath = s"""- field (class: "$clsName", name: "$fieldName")""" +: walkedTypePath

0 commit comments

Comments
 (0)