You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/spark/optimize-write-for-apache-spark.md
+3-4Lines changed: 3 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,8 +19,7 @@ Optimize Write is a Delta Lake on Synapse feature that reduces the number of fil
19
19
This feature achieves the file size by using an extra data shuffle phase over partitions, causing an extra processing cost while writing the data. The small write penalty should be outweighed by read efficiency on the tables.
20
20
21
21
> [!NOTE]
22
-
> - Optimize write is available as a Preview feature.
23
-
> - It is available on Synapse Pools for Apache Spark versions 3.1 and 3.2.
22
+
> - It is available on Synapse Pools for Apache Spark versions above 3.1.
24
23
25
24
## Benefits of Optimize Writes
26
25
@@ -48,7 +47,7 @@ This feature achieves the file size by using an extra data shuffle phase over pa
48
47
49
48
## How to enable and disable the optimize write feature
50
49
51
-
The optimize write feature is disabled by default.
50
+
The optimize write feature is disabled by default. In Spark 3.3 Pool, it is enabled by default for partitioned tables.
52
51
53
52
Once the configuration is set for the pool or session, all Spark write patterns will use the functionality.
54
53
@@ -172,4 +171,4 @@ SET `spark.microsoft.delta.optimizeWrite.binSize` = 134217728
172
171
-[Use serverless Apache Spark pool in Synapse Studio](../quickstart-create-apache-spark-pool-studio.md).
173
172
-[Run a Spark application in notebook](./apache-spark-development-using-notebooks.md).
174
173
-[Create Apache Spark job definition in Azure Studio](./apache-spark-job-definitions.md).
0 commit comments