Skip to content

Commit ed20139

Browse files
rednaxelafxcloud-fan
authored andcommitted
[SPARK-49872][FOLLOWUP] Remove Jackson JSON string length limit in KVStoreScalaSerializer
### What changes were proposed in this pull request? Along the same lines as #52049, apply the same fix to `KVStoreScalaSerializer` as well. ### Why are the changes needed? The previous fix in #52049 only removed the Jackson JSON string length limit in `JsonProtocol`, which covered Spark event <=> JSON SerDe. However, the exact same problem exists in `KVStoreScalaSerializer` as well, which is triggered when SHS is configured to use LevelDB/RocksDB KVStores. Example call chain that triggers the issue: ``` ExecutionPage.render() → sqlStore.planGraph(executionId, version) → store.read(classOf[SparkPlanGraphWrapper], ...) → KVStoreSerializer.deserialize() → mapper.readValue() // ← FAILS HERE with 20MB limit ``` ### Does this PR introduce _any_ user-facing change? Yes, users won't hit the Jackson JSON string length limit any more when using SHS. ### How was this patch tested? Manually tested with a SQL workload that had an intentionally long string for the plan graph. The fix mechanism itself is tested by previous PRs #49163 and #52049, this is just applying the same fix to SHS as well. ### Was this patch authored or co-authored using generative AI tooling? No Closes #53711 from rednaxelafx/fix-jackson-strlenlimit-kvstore. Authored-by: Kris Mok <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent ce6b5a6 commit ed20139

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

core/src/main/scala/org/apache/spark/status/KVUtils.scala

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ import scala.jdk.CollectionConverters._
2525
import scala.reflect.{classTag, ClassTag}
2626

2727
import com.fasterxml.jackson.annotation.JsonInclude
28+
import com.fasterxml.jackson.core.StreamReadConstraints
2829
import com.fasterxml.jackson.module.scala.DefaultScalaModule
2930
import org.fusesource.leveldbjni.internal.NativeDB
3031
import org.rocksdb.RocksDBException
@@ -76,6 +77,10 @@ private[spark] object KVUtils extends Logging {
7677
mapper.registerModule(DefaultScalaModule)
7778
mapper.setDefaultPropertyInclusion(JsonInclude.Include.NON_ABSENT)
7879

80+
// SPARK-49872: Remove jackson JSON string length limitation.
81+
mapper.getFactory.setStreamReadConstraints(
82+
StreamReadConstraints.builder().maxStringLength(Int.MaxValue).build()
83+
)
7984
}
8085

8186
/**

0 commit comments

Comments
 (0)