Merge pull request #234201 from sezruby/lsmdoc

Low Shuffle Merge doc

Author: v-shils

articles/synapse-analytics/spark/low-shuffle-merge-for-apache-spark.md

@@ -0,0 +1,88 @@
+---
+title: Low Shuffle Merge optimization on Delta tables
+description: Low Shuffle Merge optimization on Delta tables for Apache Spark
+author: sezruby
+ms.service: synapse-analytics 
+ms.topic: reference
+ms.subservice: spark
+ms.date: 04/11/2023
+ms.author: eunjinsong
+ms.reviewer: dacoelho
+---
+
+# Low Shuffle Merge optimization on Delta tables
+
+Delta Lake [MERGE command](https://docs.delta.io/latest/delta-update.html#upsert-into-a-table-using-merge) allows users to update a delta table with advanced conditions. It can update data from a source table, view or DataFrame into a target table by using MERGE command. However, the current algorithm isn't fully optimized for handling *unmodified* rows. With Low Shuffle Merge optimization, unmodified rows are excluded from an expensive shuffling operation that is needed for updating matched rows.
+
+## Why we need Low Shuffle Merge
+
+Currently MERGE operation is done by two Join executions. The first join is using the whole target table and source data, to find a list of *touched* files of the target table including any matched rows. After that, it performs the second join reading only those *touched* files and source data, to do actual table update. Even though the first join is to reduce the amount of data for the second join, there could still be a huge number of *unmodified* rows in *touched* files. The first join query is lighter as it only reads columns in the given matching condition. The second one for table update needs to load all columns, which incurs an expensive shuffling process.
+
+With Low Shuffle Merge optimization, Delta keeps the matched row result from the first join temporarily and utilizes it for the second join. Based on the result, it excludes *unmodified* rows from the heavy shuffling process. There would be two separate write jobs for *matched* rows and *unmodified* rows, thus it could result in 2x number of output files compared to the previous behavior. However, the expected performance gain outweighs the possible small files problem.
+
+## Availability
+
+> [!NOTE]
+> - Low Shuffle Merge is available as a Preview feature. 
+
+It's available on Synapse Pools for Apache Spark versions 3.2 and 3.3.
+
+|Version| Availability | Default |
+|--|--|--|
+| Delta 0.6 / Spark 2.4 | No | - |
+| Delta 1.2 / Spark 3.2 | Yes | false |
+| Delta 2.2 / Spark 3.3 | Yes | true |
+
+
+## Benefits of Low Shuffle Merge
+
+* Unmodified rows in *touched* files are handled separately and not going through the actual MERGE operation. It can save the overall MERGE execution time and compute resources. The gain would be larger when many rows are copied and only a few rows are updated.
+* Row orderings are preserved for unmodified rows. Therefore, the output files of unmodified rows could be still efficient for data skipping if the file was sorted or Z-ORDERED.
+* There would be tiny overhead even for the worst case when MERGE condition matches all rows in touched files.
+
+
+## How to enable and disable Low Shuffle Merge
+
+Once the configuration is set for the pool or session, all Spark write patterns will use the functionality.
+
+To use Low Shuffle Merge optimization, enable it using the following configuration:
+
+1. Scala and PySpark
+
+```scala
+spark.conf.set("spark.microsoft.delta.merge.lowShuffle.enabled", "true")
+```
+
+2. Spark SQL
+
+```SQL
+SET `spark.microsoft.delta.merge.lowShuffle.enabled` = true
+```
+
+To check the current configuration value, use the command as shown below:
+
+1. Scala and PySpark
+
+```scala
+spark.conf.get("spark.microsoft.delta.merge.lowShuffle.enabled")
+```
+
+2. Spark SQL
+
+```SQL
+SET `spark.microsoft.delta.merge.lowShuffle.enabled`
+```
+
+To disable the feature, change the following configuration as shown below:
+
+1. Scala and PySpark
+
+```scala
+spark.conf.set("spark.microsoft.delta.merge.lowShuffle.enabled", "false")
+```
+
+2. Spark SQL
+
+```SQL
+SET `spark.microsoft.delta.merge.lowShuffle.enabled` = false
+```

articles/synapse-analytics/spark/optimize-write-for-apache-spark.md

@@ -19,8 +19,7 @@ Optimize Write is a Delta Lake on Synapse feature that reduces the number of fil
 This feature achieves the file size by using an extra data shuffle phase over partitions, causing an extra processing cost while writing the data. The small write penalty should be outweighed by read efficiency on the tables.
 
 > [!NOTE]
-> - Optimize write is available as a Preview feature. 
-> - It is available on Synapse Pools for Apache Spark versions 3.1 and 3.2.
+> - It is available on Synapse Pools for Apache Spark versions above 3.1.
 
 ## Benefits of Optimize Writes
 
@@ -48,7 +47,7 @@ This feature achieves the file size by using an extra data shuffle phase over pa
 
 ## How to enable and disable the optimize write feature
 
-The optimize write feature is disabled by default.
+The optimize write feature is disabled by default. In Spark 3.3 Pool, it is enabled by default for partitioned tables.
 
 Once the configuration is set for the pool or session, all Spark write patterns will use the functionality.
 
@@ -172,4 +171,4 @@ SET `spark.microsoft.delta.optimizeWrite.binSize` = 134217728
  - [Use serverless Apache Spark pool in Synapse Studio](../quickstart-create-apache-spark-pool-studio.md).
  - [Run a Spark application in notebook](./apache-spark-development-using-notebooks.md).
  - [Create Apache Spark job definition in Azure Studio](./apache-spark-job-definitions.md).
- 
+ 

articles/synapse-analytics/toc.yml

@@ -722,6 +722,8 @@ items:
       href: ./spark/apache-spark-what-is-delta-lake.md
     - name: Optimize Apache Spark writes on Delta Lake
       href: ./spark/optimize-write-for-apache-spark.md
+    - name: Low Shuffle Merge on Delta Lake
+      href: ./spark/low-shuffle-merge-for-apache-spark.md 
     - name: Apache Spark autoscale behavior
       href: ./spark/apache-spark-autoscale.md
     - name: Intelligent Cache