Skip to content

Commit 85a9359

Browse files
mgaido91gatorsmile
authored andcommitted
[SPARK-25609][TESTS] Reduce time of test for SPARK-22226
## What changes were proposed in this pull request? The PR changes the test introduced for SPARK-22226, so that we don't run analysis and optimization on the plan. The scope of the test is code generation and running the above mentioned operation is expensive and useless for the test. The UT was also moved to the `CodeGenerationSuite` which is a better place given the scope of the test. ## How was this patch tested? running the UT before SPARK-22226 fails, after it passes. The execution time is about 50% the original one. On my laptop this means that the test now runs in about 23 seconds (instead of 50 seconds). Closes apache#22629 from mgaido91/SPARK-25609. Authored-by: Marco Gaido <[email protected]> Signed-off-by: gatorsmile <[email protected]>
1 parent 3ae4f07 commit 85a9359

File tree

2 files changed

+10
-12
lines changed

2 files changed

+10
-12
lines changed

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -346,6 +346,16 @@ class CodeGenerationSuite extends SparkFunSuite with ExpressionEvalHelper {
346346
projection(row)
347347
}
348348

349+
test("SPARK-22226: splitExpressions should not generate codes beyond 64KB") {
350+
val colNumber = 10000
351+
val attrs = (1 to colNumber).map(colIndex => AttributeReference(s"_$colIndex", IntegerType)())
352+
val lit = Literal(1000)
353+
val exprs = attrs.flatMap { a =>
354+
Seq(If(lit < a, lit, a), sqrt(a))
355+
}
356+
UnsafeProjection.create(exprs, attrs)
357+
}
358+
349359
test("SPARK-22543: split large predicates into blocks due to JVM code size limit") {
350360
val length = 600
351361

sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2408,18 +2408,6 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
24082408
Seq(Row(7, 1, 1), Row(7, 1, 2), Row(7, 2, 1), Row(7, 2, 2), Row(7, 3, 1), Row(7, 3, 2)))
24092409
}
24102410

2411-
test("SPARK-22226: splitExpressions should not generate codes beyond 64KB") {
2412-
val colNumber = 10000
2413-
val input = spark.range(2).rdd.map(_ => Row(1 to colNumber: _*))
2414-
val df = sqlContext.createDataFrame(input, StructType(
2415-
(1 to colNumber).map(colIndex => StructField(s"_$colIndex", IntegerType, false))))
2416-
val newCols = (1 to colNumber).flatMap { colIndex =>
2417-
Seq(expr(s"if(1000 < _$colIndex, 1000, _$colIndex)"),
2418-
expr(s"sqrt(_$colIndex)"))
2419-
}
2420-
df.select(newCols: _*).collect()
2421-
}
2422-
24232411
test("SPARK-22271: mean overflows and returns null for some decimal variables") {
24242412
val d = 0.034567890
24252413
val df = Seq(d, d, d, d, d, d, d, d, d, d).toDF("DecimalCol")

0 commit comments

Comments
 (0)