Delta Tables in Amazon S3 destination connector: new output format (#674)

Author: Paul-Cornell

snippets/general-shared-text/delta-table.mdx

@@ -10,7 +10,7 @@ allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; pic
 allowfullscreen
 ></iframe>
 
-The preceding video does not show how to create an AWS account or an S3 bucket. 
+The preceding video does not show how to create an AWS account. 
 
 For more information about requirements, see the following:
 
@@ -88,4 +88,27 @@ import S3BucketCloudFormation from '/snippets/general-shared-text/s3-cf-setup.md
 
 import S3BucketCLI from '/snippets/general-shared-text/s3-cli-setup.mdx';
 
-<S3BucketCLI />
+<S3BucketCLI />
+
+## Delta table output format
+
+A Delta table consists of Parquet files that contain data and a transaction log that stores metadata about the transactions. 
+[Learn more](https://delta-io.github.io/delta-rs/how-delta-lake-works/architecture-of-delta-table/).
+
+The Delta Tables in Amazon S3 destination connector generates the following output within the specified path to the S3 bucket (or the specified folder within the bucket):
+
+- Initially, one Parquet (`.parquet`) file per file in the source location. For example, for a file in the source location named `my-file.pdf`, an associated 
+  file with the extension `.parquet` is generated. Various kinds of file transactions can result in additional Parquet files being generated. These Parquet filenames are automatically generated by the Delta Lake engine and are not meant to be manually modified.
+- A folder named `_delta_log` that contains metadata and change history about the `.parquet` files. As Parquet files are added to, changed, or removed from 
+  the specified bucket or folder path, the `_delta_log` folder is updated with any related metadata and change history details. 
+
+Together, this set of Parquet files and their associated `_delta_log` folder (and its contents) describe a single, versioned Delta table. Because of this, Unstructured recommends the following usage best practices:
+
+- In the source location, each set of source files that is to be considered as a unit for change management purposes should be controlled by a unique, dedicated 
+  Delta Tables in S3 destination connector. This connector should reference a unique, dedicated output folder within the bucket. Having 
+  multiple workflows refer to different sets of source files, yet all share the same Delta table, could results in data loss or table corruption.
+- Avoid directly modifying, adding, or deleting Parquet data files or the `_delta_log` folder within a Delta table's directory. This can lead to data loss or table corruption.
+- If you need to copy or move a Delta table to a different location, 
+  you must move or copy its entire set of Parquet files and its associated `_delta_log` folder (and its contents) together as a unit. 
+  Note that the copied or moved Delta table will 
+  no longer be controlled by the original Delta Tables in S3 destination connector.