Skip to content

Commit bbb533a

Browse files
Merge pull request #276326 from santhoshravindran7/patch-3
Update apache-spark-performance.md
2 parents 818e26b + 6f5ca6f commit bbb533a

File tree

1 file changed

+8
-1
lines changed

1 file changed

+8
-1
lines changed

articles/synapse-analytics/spark/apache-spark-performance.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,13 @@ Spark provides its own native caching mechanisms, which can be used through diff
5757
Spark operates by placing data in memory, so managing memory resources is a key aspect of optimizing the execution of Spark jobs. There are several techniques you can apply to use your cluster's memory efficiently.
5858

5959
* Prefer smaller data partitions and account for data size, types, and distribution in your partitioning strategy.
60-
* Consider the newer, more efficient [Kryo data serialization](https://github.com/EsotericSoftware/kryo), rather than the default Java serialization.
60+
* In Synapse Spark (Runtime 3.1 or higher), Kryo data serialization is enabled by default Kryo data serialization.
61+
* You can customize the kryoserializer buffer size using Spark configuration based on your workload requirements:
62+
63+
```scala
64+
// Set the desired property
65+
spark.conf.set("spark.kryoserializer.buffer.max", "256m")
66+
6167
* Monitor and tune Spark configuration settings.
6268

6369
For your reference, the Spark memory structure and some key executor memory parameters are shown in the next image.
@@ -172,6 +178,7 @@ MAX(AMOUNT) -> MAX(cast(AMOUNT as DOUBLE))
172178

173179
## Next steps
174180

181+
- [Learn about Azure Synapse runtimes for Apache Spark](./apache-spark-version-support.md)
175182
- [Tuning Apache Spark](https://spark.apache.org/docs/2.4.5/tuning.html)
176183
- [How to Actually Tune Your Apache Spark Jobs So They Work](https://www.slideshare.net/ilganeli/how-to-actually-tune-your-spark-jobs-so-they-work)
177184
- [Kryo Serialization](https://github.com/EsotericSoftware/kryo)

0 commit comments

Comments
 (0)