Skip to content

Commit 00771dc

Browse files
peter-tothgatorsmile
authored andcommitted
[SPARK-25816][SQL] Fix attribute resolution in nested extractors
## What changes were proposed in this pull request? Extractors are made of 2 expressions, one of them defines the the value to be extract from (called `child`) and the other defines the way of extraction (called `extraction`). In this term extractors have 2 children so they shouldn't be `UnaryExpression`s. `ResolveReferences` was changed in this commit: apache@36b826f which resulted a regression with nested extractors. An extractor need to define its children as the set of both `child` and `extraction`; and should try to resolve both in `ResolveReferences`. This PR changes `UnresolvedExtractValue` to a `BinaryExpression`. ## How was this patch tested? added UT Closes apache#22817 from peter-toth/SPARK-25816. Authored-by: Peter Toth <[email protected]> Signed-off-by: gatorsmile <[email protected]> (cherry picked from commit ca2fca1) Signed-off-by: gatorsmile <[email protected]>
1 parent 0f74bac commit 00771dc

File tree

2 files changed

+11
-1
lines changed

2 files changed

+11
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,10 @@ case class ResolvedStar(expressions: Seq[NamedExpression]) extends Star with Une
407407
* can be key of Map, index of Array, field name of Struct.
408408
*/
409409
case class UnresolvedExtractValue(child: Expression, extraction: Expression)
410-
extends UnaryExpression with Unevaluable {
410+
extends BinaryExpression with Unevaluable {
411+
412+
override def left: Expression = child
413+
override def right: Expression = extraction
411414

412415
override def dataType: DataType = throw new UnresolvedException(this, "dataType")
413416
override def foldable: Boolean = throw new UnresolvedException(this, "foldable")

sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2590,4 +2590,11 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
25902590
Row ("abc", 1))
25912591
}
25922592
}
2593+
2594+
test("SPARK-25816 ResolveReferences works with nested extractors") {
2595+
val df = Seq((1, Map(1 -> "a")), (2, Map(2 -> "b"))).toDF("key", "map")
2596+
val swappedDf = df.select($"key".as("map"), $"map".as("key"))
2597+
2598+
checkAnswer(swappedDf.filter($"key"($"map") > "a"), Row(2, Map(2 -> "b")))
2599+
}
25932600
}

0 commit comments

Comments
 (0)