You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: snippets/general-shared-text/delta-table.mdx
+14-10Lines changed: 14 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,8 +26,8 @@ For more information about requirements, see the following:
26
26
allowfullscreen
27
27
></iframe>
28
28
29
-
- An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
30
-
Additional approaches are in the following video and in the how-to sections at the end of this page.
29
+
- An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
30
+
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.
31
31
32
32
<iframe
33
33
width="560"
@@ -39,8 +39,15 @@ For more information about requirements, see the following:
39
39
allowfullscreen
40
40
></iframe>
41
41
42
-
- For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).
42
+
- For authenticated bucket write access or both, you should first
43
+
[block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).
43
44
45
+
After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
46
+
For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. To grant permissions,
47
+
attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
48
+
To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
49
+
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.
50
+
44
51
<iframe
45
52
width="560"
46
53
height="315"
@@ -51,11 +58,8 @@ For more information about requirements, see the following:
51
58
allowfullscreen
52
59
></iframe>
53
60
54
-
- For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
55
-
minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).
56
-
57
-
- For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
58
-
[Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).
61
+
- Provide an AWS access key and secret access key for the authenticated AWS IAM user in the account.
62
+
Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.
59
63
60
64
<iframe
61
65
width="560"
@@ -67,10 +71,10 @@ For more information about requirements, see the following:
67
71
allowfullscreen
68
72
></iframe>
69
73
70
-
- If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
74
+
- If the target files are in the root of the bucket, provide the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
71
75
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).
72
76
- If the target files are in a folder, make sure the authenticated AWS IAM user has
73
-
authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).
77
+
authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).
Copy file name to clipboardExpand all lines: snippets/general-shared-text/s3.mdx
+53-58Lines changed: 53 additions & 58 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ allowfullscreen
11
11
></iframe>
12
12
13
13
The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
14
-
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
14
+
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
15
15
requirements. For more information about requirements, see the following:
16
16
17
17
- An AWS account. [Create an AWS account](https://aws.amazon.com/free).
@@ -26,8 +26,8 @@ requirements. For more information about requirements, see the following:
26
26
allowfullscreen
27
27
></iframe>
28
28
29
-
- An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
30
-
Additional approaches are in the following video and in the how-to sections at the end of this page.
29
+
- An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
30
+
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.
31
31
32
32
<iframe
33
33
width="560"
@@ -39,61 +39,44 @@ requirements. For more information about requirements, see the following:
39
39
allowfullscreen
40
40
></iframe>
41
41
42
-
- Anonymous (supported but not recommended) or authenticated access to the bucket.
42
+
- Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
43
+
[in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.
- For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).
- For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
65
-
minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).
66
-
67
-
- For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
68
-
[Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).
- For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).
After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
51
+
For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
52
+
can be granted in one of the following ways:
53
+
54
+
- Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
55
+
To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
56
+
Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.
- Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
69
+
Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
70
+
[create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
71
+
[have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
72
+
a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).
73
+
74
+
<Warning>
75
+
AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
76
+
generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
77
+
You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
78
+
a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.
91
79
92
-
<Warning>
93
-
AWS STS credentials can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
94
-
generated. After the expiry time, the credentials are no longer valid will no longer work with the corresponding S3 connector.
95
-
You must get a new set of credentials to replace the expired ones by calling
96
-
[GetSessionToken](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html) in the AWS STS API.
97
80
To overwrite the expired credentials with the new set:
98
81
99
82
- For the Unstructured user interface (UI), manually update the AWS Key, AWS Secret Key, and STS Token fields in the Unstructured UI
@@ -105,10 +88,22 @@ requirements. For more information about requirements, see the following:
- For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
107
90
corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
91
+
</Warning>
108
92
109
-
</Warning>
110
-
111
-
- If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
93
+
- If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
94
+
Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.
- If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
112
107
If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).
113
108
- If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
114
-
authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).
109
+
authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).
0 commit comments