Skip to content

Commit 4968f87

Browse files
wangyumHyukjinKwon
authored andcommitted
[SPARK-23263][TEST] CTAS should update stat if autoUpdate statistics is enabled
## What changes were proposed in this pull request? The [SPARK-27403](https://issues.apache.org/jira/browse/SPARK-27403) fixed CTAS cannot update statistics even if `spark.sql.statistics.size.autoUpdate.enabled` is enabled, as mentioned in [SPARK-23263](https://issues.apache.org/jira/browse/SPARK-23263). This pr adds tests for that fix. ## How was this patch tested? N/A Closes apache#20430 from wangyum/SPARK-23263. Authored-by: Yuming Wang <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent f510761 commit 4968f87

File tree

1 file changed

+22
-0
lines changed

1 file changed

+22
-0
lines changed

sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1431,4 +1431,26 @@ class StatisticsSuite extends StatisticsCollectionTestBase with TestHiveSingleto
14311431
assert(catalogStats.rowCount.isEmpty)
14321432
}
14331433
}
1434+
1435+
test(s"CTAS should update statistics if ${SQLConf.AUTO_SIZE_UPDATE_ENABLED.key} is enabled") {
1436+
val tableName = "SPARK_23263"
1437+
Seq(false, true).foreach { isConverted =>
1438+
Seq(false, true).foreach { updateEnabled =>
1439+
withSQLConf(
1440+
SQLConf.AUTO_SIZE_UPDATE_ENABLED.key -> updateEnabled.toString,
1441+
HiveUtils.CONVERT_METASTORE_PARQUET.key -> isConverted.toString) {
1442+
withTable(tableName) {
1443+
sql(s"CREATE TABLE $tableName STORED AS parquet AS SELECT 'a', 'b'")
1444+
val catalogTable = getCatalogTable(tableName)
1445+
// Hive serde tables always update statistics by Hive metastore
1446+
if (!isConverted || updateEnabled) {
1447+
assert(catalogTable.stats.nonEmpty)
1448+
} else {
1449+
assert(catalogTable.stats.isEmpty)
1450+
}
1451+
}
1452+
}
1453+
}
1454+
}
1455+
}
14341456
}

0 commit comments

Comments
 (0)