Commit dd497fb
committed
[SPARK-52597][SS][TESTS] Fix the execution failure of
### What changes were proposed in this pull request?
This pr has made the following changes to fix the `StateStoreBasicOperationsBenchmark`:
1. Following the suggestion from zecookiez, set "spark.sql.streaming.stateStore.coordinatorReportSnapshotUploadLag" to false when initializing `StateStoreConf`.
2. When initializing the `RocksDBStateStoreProvider`, populate the `StreamExecution.RUN_ID_KEY` for the incoming Hadoop `Configuration`.
### Why are the changes needed?
Fix the execution failure of `StateStoreBasicOperationsBenchmark`:
```
build/sbt "sql/Test/runMain org.apache.spark.sql.execution.benchmark.StateStoreBasicOperationsBenchmark"
[error] Exception in thread "main" java.lang.AssertionError: assertion failed
[error] at scala.Predef$.assert(Predef.scala:264)
[error] at org.apache.spark.sql.execution.streaming.state.StateStoreProvider$.getRunId(StateStore.scala:673)
[error] at org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider.init(RocksDBStateStoreProvider.scala:394)
[error] at org.apache.spark.sql.execution.benchmark.StateStoreBasicOperationsBenchmark$.newRocksDBStateProvider(StateStoreBasicOperationsBenchmark.scala:484)
[error] at org.apache.spark.sql.execution.benchmark.StateStoreBasicOperationsBenchmark$.$anonfun$runPutBenchmark$3(StateStoreBasicOperationsBenchmark.scala:92)
[error] at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)
[error] at scala.collection.immutable.List.foreach(List.scala:334)
[error] at org.apache.spark.sql.execution.benchmark.StateStoreBasicOperationsBenchmark$.$anonfun$runPutBenchmark$2(StateStoreBasicOperationsBenchmark.scala:87)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error] at org.apache.spark.benchmark.BenchmarkBase.runBenchmark(BenchmarkBase.scala:42)
[error] at org.apache.spark.sql.execution.benchmark.StateStoreBasicOperationsBenchmark$.runPutBenchmark(StateStoreBasicOperationsBenchmark.scala:83)
[error] at org.apache.spark.sql.execution.benchmark.StateStoreBasicOperationsBenchmark$.runBenchmarkSuite(StateStoreBasicOperationsBenchmark.scala:55)
[error] at org.apache.spark.benchmark.BenchmarkBase.main(BenchmarkBase.scala:72)
[error] at org.apache.spark.sql.execution.benchmark.StateStoreBasicOperationsBenchmark.main(StateStoreBasicOperationsBenchmark.scala)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- It has been locally confirmed that the `StateStoreBasicOperationsBenchmark` can be executed successfully.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #51304 from LuciferYang/SPARK-52597.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>StateStoreBasicOperationsBenchmark
1 parent 16c0ce9 commit dd497fb
File tree
1 file changed
+9
-1
lines changed- sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark
1 file changed
+9
-1
lines changedLines changed: 9 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| 28 | + | |
26 | 29 | | |
27 | 30 | | |
28 | 31 | | |
| |||
477 | 480 | | |
478 | 481 | | |
479 | 482 | | |
| 483 | + | |
| 484 | + | |
480 | 485 | | |
481 | 486 | | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
482 | 490 | | |
483 | 491 | | |
484 | | - | |
| 492 | + | |
485 | 493 | | |
486 | 494 | | |
487 | 495 | | |
| |||
0 commit comments