S3 connectors: if using AWS STS, must specify access key, secret access key, and STS token (not just STS token) (#599)

Author: Paul-Cornell

snippets/destination_connectors/s3.sh.mdx

@@ -16,5 +16,6 @@ unstructured-ingest \
   s3 \
     --remote-url $AWS_S3_URL \
     --key $AWS_ACCESS_KEY_ID \
-    --secret $AWS_SECRET_ACCESS_KEY
+    --secret $AWS_SECRET_ACCESS_KEY \
+    --token $AWS_STS_TOKEN # If using AWS STS token.
 ```

snippets/destination_connectors/s3.v2.py.mdx

@@ -41,7 +41,8 @@ if __name__ == "__main__":
         destination_connection_config=S3ConnectionConfig(
             access_config=S3AccessConfig(
                 key=os.getenv("AWS_ACCESS_KEY_ID"),
-                secret=os.getenv("AWS_SECRET_ACCESS_KEY")
+                secret=os.getenv("AWS_SECRET_ACCESS_KEY"),
+                token=os.getenv("AWS_STS_TOKEN", None), # If using AWS STS token.
             )
         ),
         uploader_config=S3UploaderConfig(remote_url=os.getenv("AWS_S3_URL"))

snippets/destination_connectors/s3_rest_create.mdx

@@ -15,6 +15,8 @@ curl --request 'POST' --location \
 
         # For AWS STS token authentication:
         "token": "<token>", 
+        "key": "<key>", 
+        "secret": "<secret>", 
 
         "remote_url": "<remote_url>",
         "endpoint_url": "<endpoint-url>",

snippets/destination_connectors/s3_sdk.mdx

@@ -21,6 +21,8 @@ with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as clien
                     secret="<secret>", 
 
                     # For AWS STS token authentication:
+                    key="<key>", 
+                    secret="<secret>",
                     token="<token>", 
 
                     remote_url="<remote_url>",

snippets/general-shared-text/s3-api-placeholders.mdx

@@ -1,5 +1,5 @@
 - `<name>` (_required_) - A unique name for this connector.
-- For AWS access key ID with AWS secret access key authentication:
+- For AWS access key ID with AWS secret access key authentication, or for AWS STS token authentication:
 
   - `<key>` - The AWS access key ID for the authenticated AWS IAM user (_required_).
   - `<secret>` - The AWS secret access key corresponding to the preceding AWS access key ID (_required_).

snippets/general-shared-text/s3-cli-api.mdx

@@ -15,6 +15,6 @@ The following environment variables:
 
   - `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--key` (CLI) or `key` (Python).
   - `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--secret` (CLI) or `secret` (Python).
-  - `AWS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).
+  - `AWS_STS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).
 
 - If the bucket has anonymous access enabled for reading from the bucket, set `--anonymous` (CLI) or `anonymous=True` (Python) instead.

snippets/general-shared-text/s3-platform.mdx

@@ -3,7 +3,7 @@ Fill in the following fields:
 - **Name** (_required_): A unique name for this connector.
 - **Bucket URI** (_required_): The URI for the bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.
 - **Recursive** (source connector only): Check this box to access subfolders within the bucket.
-- **AWS Key**: For secret authentication, the AWS access key ID for the authenticated AWS IAM user.
-- **AWS Secret Key**: For secret authentication, the AWS secret access key corresponding to the preceding AWS access key ID.
-- **Token**: For token authentication, the AWS STS session token for temporary access.
+- **AWS Key**: For secret or token authentication, the AWS access key ID for the authenticated AWS IAM user.
+- **AWS Secret Key**: For secret or token authentication, the AWS secret access key corresponding to the preceding AWS access key ID.
+- **STS Token**: For token authentication, the AWS STS session token for temporary access.
 - **Custom URL**: A custom URL, if connecting to a non-AWS S3 bucket.

snippets/source_connectors/s3.sh.mdx

@@ -9,6 +9,7 @@ unstructured-ingest \
     --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
     --key $AWS_ACCESS_KEY_ID \
     --secret $AWS_SECRET_ACCESS_KEY \
+    --token $AWS_STS_TOKEN \ # If using AWS STS token.
     --partition-by-api \
     --api-key $UNSTRUCTURED_API_KEY \
     --partition-endpoint $UNSTRUCTURED_API_URL \

snippets/source_connectors/s3.v2.py.mdx

@@ -24,7 +24,8 @@ if __name__ == "__main__":
         source_connection_config=S3ConnectionConfig(
             access_config=S3AccessConfig(
                 key=os.getenv("AWS_ACCESS_KEY_ID"),
-                secret=os.getenv("AWS_SECRET_ACCESS_KEY")
+                secret=os.getenv("AWS_SECRET_ACCESS_KEY"),
+                token=os.getenv("AWS_STS_TOKEN", None)  # If using AWS STS token.
             )
         ),
         partitioner_config=PartitionerConfig(

snippets/source_connectors/s3_rest_create.mdx

@@ -18,6 +18,8 @@ curl --request 'POST' --location \
 
         # For AWS STS token authentication:
         "token": "<token>", 
+        "key": "<key>", 
+        "secret": "<secret>", 
 
         "remote_url": "<remote_url>",
         "endpoint_url": "<endpoint-url>",