Merge pull request #120948 from kim-ale/patch-7

Add AddFileAction Size Limitation

Author: Court72

articles/stream-analytics/write-to-delta-lake.md

@@ -84,8 +84,7 @@ At the failure of schema conversion, the job behavior will follow the [output da
 
 ### Delta Log checkpoints
 
-
-The Stream Analytics job will create Delta Log checkpoints periodically.
+The Stream Analytics job will create [Delta Log checkpoints](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#checkpoints-1) periodically in the V1 format. Delta Log checkpoints are snapshots of the Delta Table and will typically contain the name of the data file generated by the Stream Analytics job. If the amount of data files is large, then this will lead to large checkpoints which can cause memory issues in the Stream Analytics Job.
 
 ## Limitations
 
@@ -100,6 +99,14 @@ The Stream Analytics job will create Delta Log checkpoints periodically.
 - Writing to existing tables of Writer Version 7 or above with writer features will fail.
     - Example: Writing to existing tables with [Deletion Vectors](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#deletion-vectors) enabled will fail.
     - The exceptions here are the [changeDataFeed and appendOnly Writer Features](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#valid-feature-names-in-table-features).
+- When a Stream Analytics job writes a batch of data to a Delta Lake, it can generate multiple [Add File Action](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file). When there are too many Add File Actions generated for a single batch, a Stream Analytics Job can be stuck.
+    - The number of Add File Actions generated are determined by a number of factors:
+        - Size of the batch. This is determined by the data volume and the batching parameters [Minimum Rows and Maximum Time](https://learn.microsoft.com/azure/stream-analytics/blob-storage-azure-data-lake-gen2-output#output-configuration)
+        - Cardinality of the [Partition Column values](https://learn.microsoft.com/azure/stream-analytics/write-to-delta-lake#delta-lake-configuration) of the batch.
+    - To reduce the number of Add File Actions generated for a batch the following steps can be taken:
+        - Reduce the batching configurations [Minimum Rows and Maximum Time](https://learn.microsoft.com/azure/stream-analytics/blob-storage-azure-data-lake-gen2-output#output-configuration)
+        - Reduce the cardinality of the [Partition Column values](https://learn.microsoft.com/azure/stream-analytics/write-to-delta-lake#delta-lake-configuration) by tweaking the input data or choosing a different partition column
+- Stream Analytics jobs can only read and write single part V1 Checkpoints. Multi-part checkpoints and the Checkpoint V2 format are not supported.
 
 ## Next steps