Skip to content

Commit 06ac7d5

Browse files
manu.zhangHyukjinKwon
authored andcommitted
[SPARK-27922][SQL][PYTHON][TESTS] Convert and port 'natural-join.sql' into UDF test base
## What changes were proposed in this pull request? This PR adds some tests converted from `natural-join.sql` to test UDFs following the combination guide in [SPARK-27921](https://issues.apache.org/jira/browse/SPARK-27921). <details><summary>Diff results comparing to `natural-join.sql`</summary> <p> ```diff diff --git a/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join.sql.out b/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join. sql.out index 43f2f9a..53ef177 100644 --- a/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join.sql.out -27,7 +27,7 struct<> -- !query 2 -SELECT * FROM nt1 natural join nt2 where k = "one" +SELECT * FROM nt1 natural join nt2 where udf(k) = "one" -- !query 2 schema struct<k:string,v1:int,v2:int> -- !query 2 output -36,7 +36,7 one 1 5 -- !query 3 -SELECT * FROM nt1 natural left join nt2 order by v1, v2 +SELECT * FROM nt1 natural left join nt2 where k <> udf("") order by v1, v2 -- !query 3 schema diff --git a/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join.sql.out b/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join. sql.out index 43f2f9a..53ef177 100644 --- a/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/udf/udf-natural-join.sql.out -27,7 +27,7 struct<> -- !query 2 -SELECT * FROM nt1 natural join nt2 where k = "one" +SELECT * FROM nt1 natural join nt2 where udf(k) = "one" -- !query 2 schema struct<k:string,v1:int,v2:int> -- !query 2 output -36,7 +36,7 one 1 5 -- !query 3 -SELECT * FROM nt1 natural left join nt2 order by v1, v2 +SELECT * FROM nt1 natural left join nt2 where k <> udf("") order by v1, v2 -- !query 3 schema struct<k:string,v1:int,v2:int> ``` </p> </details> ## How was this patch tested? Tested as guided in [SPARK-27921](https://issues.apache.org/jira/browse/SPARK-27921). Closes apache#25088 from manuzhang/SPARK-27922. Authored-by: manu.zhang <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent f84cca2 commit 06ac7d5

File tree

2 files changed

+93
-0
lines changed

2 files changed

+93
-0
lines changed
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
-- List of configuration the test suite is run against:
2+
--SET spark.sql.autoBroadcastJoinThreshold=10485760
3+
--SET spark.sql.autoBroadcastJoinThreshold=-1,spark.sql.join.preferSortMergeJoin=true
4+
--SET spark.sql.autoBroadcastJoinThreshold=-1,spark.sql.join.preferSortMergeJoin=false
5+
6+
-- This test file was converted from natural-join.sql.
7+
-- Note that currently registered UDF returns a string. So there are some differences, for instance
8+
-- in string cast within UDF in Scala and Python.
9+
10+
create temporary view nt1 as select * from values
11+
("one", 1),
12+
("two", 2),
13+
("three", 3)
14+
as nt1(k, v1);
15+
16+
create temporary view nt2 as select * from values
17+
("one", 1),
18+
("two", 22),
19+
("one", 5)
20+
as nt2(k, v2);
21+
22+
23+
SELECT * FROM nt1 natural join nt2 where udf(k) = "one";
24+
25+
SELECT * FROM nt1 natural left join nt2 where k <> udf("") order by v1, v2;
26+
27+
SELECT * FROM nt1 natural right join nt2 where udf(k) <> udf("") order by v1, v2;
28+
29+
SELECT udf(count(*)) FROM nt1 natural full outer join nt2;
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
-- Automatically generated by SQLQueryTestSuite
2+
-- Number of queries: 6
3+
4+
5+
-- !query 0
6+
create temporary view nt1 as select * from values
7+
("one", 1),
8+
("two", 2),
9+
("three", 3)
10+
as nt1(k, v1)
11+
-- !query 0 schema
12+
struct<>
13+
-- !query 0 output
14+
15+
16+
17+
-- !query 1
18+
create temporary view nt2 as select * from values
19+
("one", 1),
20+
("two", 22),
21+
("one", 5)
22+
as nt2(k, v2)
23+
-- !query 1 schema
24+
struct<>
25+
-- !query 1 output
26+
27+
28+
29+
-- !query 2
30+
SELECT * FROM nt1 natural join nt2 where udf(k) = "one"
31+
-- !query 2 schema
32+
struct<k:string,v1:int,v2:int>
33+
-- !query 2 output
34+
one 1 1
35+
one 1 5
36+
37+
38+
-- !query 3
39+
SELECT * FROM nt1 natural left join nt2 where k <> udf("") order by v1, v2
40+
-- !query 3 schema
41+
struct<k:string,v1:int,v2:int>
42+
-- !query 3 output
43+
one 1 1
44+
one 1 5
45+
two 2 22
46+
three 3 NULL
47+
48+
49+
-- !query 4
50+
SELECT * FROM nt1 natural right join nt2 where udf(k) <> udf("") order by v1, v2
51+
-- !query 4 schema
52+
struct<k:string,v1:int,v2:int>
53+
-- !query 4 output
54+
one 1 1
55+
one 1 5
56+
two 2 22
57+
58+
59+
-- !query 5
60+
SELECT udf(count(*)) FROM nt1 natural full outer join nt2
61+
-- !query 5 schema
62+
struct<udf(count(1)):string>
63+
-- !query 5 output
64+
4

0 commit comments

Comments
 (0)