Skip to content

Commit ee2647e

Browse files
LuciferYangsrowen
LuciferYang
authored and
srowen
committed
[SPARK-37008][SQL][TEST] Replace UseCompressedOops with UseCompressedClassPointers to pass WholeStageCodegenSparkSubmitSuite with Java 17
### What changes were proposed in this pull request? `UseCompressedOops` Option has some different behaviors between Java 8, Java 11 and Java 17, if we execute command ``` java -XX:+PrintFlagsFinal -XX:-UseCompressedOops -version | grep Compressed ``` The result of Java 8 is ``` bool UseCompressedClassPointers = false {lp64_product} bool UseCompressedOops := false {lp64_product} ``` The result of Java 11 is ``` bool UseCompressedClassPointers = false {lp64_product} {default} bool UseCompressedOops = false {lp64_product} {command line} ``` The result of Java 17 is ``` bool UseCompressedClassPointers = true {product lp64_product} {ergonomic} bool UseCompressedOops = false {product lp64_product} {command line} ``` We found that `-XX:-UseCompressedOops` and `-XX:-UseCompressedClassPointers` are no longer react in chain when use Java 17, if we execute ``` mvn clean install -pl sql/core -am -Dtest=none -DwildcardSuites=org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite ``` with Java 17, the test failed due to `Exception in thread "main" org.scalatest.exceptions.TestFailedException: 16 was not greater than 16`, so the change of this pr is replace `UseCompressedOops` with `UseCompressedClassPointers` to pass `WholeStageCodegenSparkSubmitSuite` with Java 17 . ### Why are the changes needed? Pass UT with JDK 17 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass the Jenkins or GitHub Action - Manual test with Java 11 passed ``` Run completed in 15 seconds, 546 milliseconds. Total number of tests run: 1 Suites: completed 2, aborted 0 Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` - Manual test with Java 17 ``` mvn clean install -pl sql/core -am -Dtest=none -DwildcardSuites=org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite ``` **Before** ``` 2021-10-14 04:32:38.038 - stderr> Exception in thread "main" org.scalatest.exceptions.TestFailedException: 16 was not greater than 16 2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) 2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) 2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) 2021-10-14 04:32:38.038 - stderr> at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite$.main(WholeStageCodegenSparkSubmitSuite.scala:82) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite.main(WholeStageCodegenSparkSubmitSuite.scala) 2021-10-14 04:32:38.038 - stderr> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2021-10-14 04:32:38.038 - stderr> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) 2021-10-14 04:32:38.038 - stderr> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2021-10-14 04:32:38.038 - stderr> at java.base/java.lang.reflect.Method.invoke(Method.java:568) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) 2021-10-14 04:32:38.038 - stderr> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` **After** ``` Run completed in 11 seconds, 15 milliseconds. Total number of tests run: 1 Suites: completed 2, aborted 0 Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes apache#34286 from LuciferYang/SPARK-37008. Authored-by: yangjie01 <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent cf43623 commit ee2647e

File tree

1 file changed

+14
-3
lines changed

1 file changed

+14
-3
lines changed

‎sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,15 +40,26 @@ class WholeStageCodegenSparkSubmitSuite extends SparkSubmitTestUtils
4040
val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
4141

4242
// HotSpot JVM specific: Set up a local cluster with the driver/executor using mismatched
43-
// settings of UseCompressedOops JVM option.
43+
// settings of UseCompressedClassPointers JVM option.
4444
val argsForSparkSubmit = Seq(
4545
"--class", WholeStageCodegenSparkSubmitSuite.getClass.getName.stripSuffix("$"),
4646
"--master", "local-cluster[1,1,1024]",
4747
"--driver-memory", "1g",
4848
"--conf", "spark.ui.enabled=false",
4949
"--conf", "spark.master.rest.enabled=false",
50-
"--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedOops",
51-
"--conf", "spark.executor.extraJavaOptions=-XX:+UseCompressedOops",
50+
// SPARK-37008: The results of `Platform.BYTE_ARRAY_OFFSET` using different Java versions
51+
// and different args as follows table:
52+
// +------------------------------+--------+---------+
53+
// | |Java 8 |Java 17 |
54+
// +------------------------------+--------+---------+
55+
// |-XX:-UseCompressedOops | 24 | 16 |
56+
// |-XX:+UseCompressedOops | 16 | 16 |
57+
// |-XX:-UseCompressedClassPointers| 24 | 24 |
58+
// |-XX:+UseCompressedClassPointers| 16 | 16 |
59+
// +-------------------------------+-------+---------+
60+
// So SPARK-37008 replace `UseCompressedOops` with `UseCompressedClassPointers`.
61+
"--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedClassPointers",
62+
"--conf", "spark.executor.extraJavaOptions=-XX:+UseCompressedClassPointers",
5263
"--conf", "spark.sql.adaptive.enabled=false",
5364
unusedJar.toString)
5465
runSparkSubmit(argsForSparkSubmit, timeout = 3.minutes)

0 commit comments

Comments
 (0)