Add AddFileAction Size Limitation

Author: kim-ale

articles/stream-analytics/write-to-delta-lake.md

@@ -84,7 +84,6 @@ At the failure of schema conversion, the job behavior will follow the [output da
 
 ### Delta Log checkpoints
 
-
 The Stream Analytics job will create Delta Log checkpoints periodically.
 
 ## Limitations
@@ -100,6 +99,13 @@ The Stream Analytics job will create Delta Log checkpoints periodically.
 - Writing to existing tables of Writer Version 7 or above with writer features will fail.
     - Example: Writing to existing tables with [Deletion Vectors](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#deletion-vectors) enabled will fail.
     - The exceptions here are the [changeDataFeed and appendOnly Writer Features](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#valid-feature-names-in-table-features).
+- When a Stream Analytics job writes a batch of data to a Delta Lake, it can generate multiple [Add File Action](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file). When there are too many Add File Actions generated for a single batch, a Stream Analytics Job can be stuck.
+    - The number of Add File Actions generated are determined by a number of factors:
+        - Size of the batch. This is determined by the data volume and the batching parameters [Minimum Rows and Maximum Time](https://learn.microsoft.com/en-us/azure/stream-analytics/blob-storage-azure-data-lake-gen2-output#output-configuration)
+        - Cardinality of the [Partition Column values](https://learn.microsoft.com/en-us/azure/stream-analytics/write-to-delta-lake#delta-lake-configuration) of the batch.
+    - To reduce the number of Add File Actions generated for a batch the following steps can be taken:
+        - Reduce the batching configurations [Minimum Rows and Maximum Time](https://learn.microsoft.com/en-us/azure/stream-analytics/blob-storage-azure-data-lake-gen2-output#output-configuration)
+        - Reduce the cardinality of the [Partition Column values](https://learn.microsoft.com/en-us/azure/stream-analytics/write-to-delta-lake#delta-lake-configuration) by tweaking the input data or choosing a different partition column
 
 ## Next steps