Skip to content

Commit ad873c2

Browse files
authored
docs: Minor improvements to Spark SQL test docs (#1980)
1 parent f7c92aa commit ad873c2

File tree

1 file changed

+8
-4
lines changed

1 file changed

+8
-4
lines changed

docs/source/contributor-guide/spark-sql-tests.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -67,16 +67,16 @@ ENABLE_COMET=true build/sbt "hive/testOnly * -- -n org.apache.spark.tags.SlowHiv
6767
```
6868
#### Steps to run individual test suites through SBT
6969
1. Open SBT with Comet enabled
70-
```sbt
71-
ENABLE_COMET=true sbt -Dspark.test.includeSlowTests=true
70+
```shell
71+
ENABLE_COMET=true sbt -J-Xmx4096m -Dspark.test.includeSlowTests=true
7272
```
7373
2. Run individual tests (Below code runs test named `SPARK-35568` in the `spark-sql` module)
74-
```sbt
74+
```shell
7575
sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn -- -z "SPARK-35568"
7676
```
7777
#### Steps to run individual test suites in IntelliJ IDE
7878
1. Add below configuration in VM Options for your test case (apache-spark repository)
79-
```sbt
79+
```shell
8080
-Dspark.comet.enabled=true -Dspark.comet.debug.enabled=true -Dspark.plugins=org.apache.spark.CometPlugin -DXmx4096m -Dspark.executor.heartbeatInterval=20000 -Dspark.network.timeout=10000 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED
8181
```
8282
2. Set `ENABLE_COMET=true` in environment variables
@@ -138,8 +138,12 @@ wiggle --replace ./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.sc
138138
The diff file can be generated using the `git diff` command. It may be necessary to set the `core.abbrev`
139139
configuration setting to use 11 digits hashes for consistency with existing diff files.
140140

141+
Note that there is an `IgnoreComet.scala` that is not part of the Spark codebase, and therefore needs to be added
142+
using `git add` before generating the diff.
143+
141144
```shell
142145
git config core.abbrev 11;
146+
git add sql/core/src/test/scala/org/apache/spark/sql/IgnoreComet.scala
143147
git diff v3.5.6 > ../datafusion-comet/dev/diffs/3.5.6.diff
144148
```
145149

0 commit comments

Comments
 (0)