You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -3,11 +3,13 @@ title: AWS S3 with IAM Role Support Destination
3
3
hide-personas-partial: true
4
4
---
5
5
6
-
{% include content/beta-note.md %}
6
+
> info "This document is about a destination which is in beta"
7
+
> This means that the AWS S3 with IAM Role Support destination is in active development, and some functionality may change before it becomes generally available.
8
+
7
9
8
10
## Getting Started
9
11
10
-
The Amazon S3 destination puts the raw logs of the data Segment receives into your S3 bucket, encrypted, no matter what region the bucket is in.
12
+
The AWS destination puts the raw logs of the data Segment receives into your S3 bucket, encrypted, no matter what region the bucket is in.
11
13
12
14
> info ""
13
15
> Segment copies data into your bucket every hour around the :40 minute mark. You may see multiple files over a period of time depending on the amount of data Segment copies.
@@ -28,25 +30,7 @@ Complete the following steps to configure the AWS S3 Destination with IAM Role S
28
30
29
31
To complete this section, you need access to your AWS dashboard.
30
32
31
-
1. Create a new S3 bucket in your preferred region. For more information, see Amazon's documentation, [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html){:target="_blank"}. Add the following policy to the bucket to allow Segment to copy files into it:
This adds the ability to `s3:PutObject` for the Segment s3-copy user for your bucket.
33
+
1. Create a new S3 bucket in your preferred region. For more information, see Amazon's documentation, [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html){:target="_blank"}.
50
34
2. Create a new IAM role for Segment to assume. For more information, see Amazon's documentation, [Creating a role to delegate permissions to an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html){:target="_blank"}.
51
35
3. Attach the following trust relationship document. Be sure to add your Workspace ID to the `sts:ExternalId` field.
52
36
```json
@@ -95,10 +79,10 @@ To complete this section, you need access to your AWS dashboard.
95
79
"Sid": "AllowKMS",
96
80
"Effect": "Allow",
97
81
"Action": [
98
-
"kms:GenerateDataKey",
99
-
"kms:Decrypt"
82
+
"kms:GenerateDataKey",
83
+
"kms:Decrypt"
100
84
],
101
-
"Resource": "YOUR_KEY_ARN"
85
+
"Resource": "<YOUR_KEY_ARN>"
102
86
}
103
87
]
104
88
}
@@ -111,7 +95,7 @@ If you have server-side encryption enabled, see the [required configuration](#en
111
95
112
96
To finish configuration, enable the AWS S3 Destination with IAM Role Support destination in your workspace.
113
97
114
-
1. Add the destination from the Data Storage catalog.
98
+
1. Add the AWS S3 destination from the Data Storage section of the Destinations catalog.
115
99
2. Select the data source you'll connect to the destination.
116
100
3. Provide a unique name for the destination.
117
101
4. Complete the destination settings:
@@ -124,18 +108,17 @@ To finish configuration, enable the AWS S3 Destination with IAM Role Support des
124
108
## Migrate an existing destination
125
109
To migrate an existing Amazon S3 destination to the AWS S3 with IAM Role Support Destination:
126
110
127
-
1. Configure the IAM role and IAM policy permissions as described in steps 3 and 4 [above](#create-an-iam-role-in-aws).
111
+
1. Configure the IAM role and IAM policy permissions as described in steps 2 - 4 [above](#create-an-iam-role-in-aws).
128
112
2. Add the AWS S3 with IAM Role Support Destination and add the AWS Region and IAM role ARN. For the bucket name, enter `<YOUR_BUCKET_NAME>/segment-logs/test`. Enable the destination, and verify data is received at `<YOUR_BUCKET_NAME>/segment-logs/test/segment-logs`. If the folder receives data, continue to the next step. If you don't see log entries, check the trust relationship document and IAM policy attached to the role.
129
113
3. Update the bucket name in the new destination to `<YOUR_BUCKET_NAME>`.
130
114
4. After 1 hour, disable the original Amazon S3 destination to avoid data duplication.
131
115
5. Verify that the `<YOUR_BUCKET_NAME>/segment-logs` receives data.
132
116
6. Remove the test folder created in step 2 from the bucket.
133
117
134
-
{% comment %}
135
-
### Migration steps for users with multiple sources per environment
136
118
137
-
In cases where users have multiple sources per environment, for example staging sources pointing to a staging bucket, and production sources going to a production bucket, they need two IAM roles, one for staging, and one for production.
119
+
### Migration steps for scenarios with multiple sources per environment
138
120
121
+
In cases where you have multiple sources per environment, for example staging sources pointing to a staging bucket, and production sources going to a production bucket, you need two IAM roles, one for staging, and one for production.
139
122
140
123
For example:
141
124
@@ -146,10 +129,9 @@ For example:
146
129
- prod_source_2 → prod_bucket
147
130
- prod_source_N → prod_bucket
148
131
149
-
In this scenario, for `stage_source_1`:
150
-
1.
132
+
For each source in the scenario, complete the steps described in [Migrate an existing destination](#migrate-an-existing-destination), and ensure that you have separate IAM Roles and Permissions set for staging and production use.
133
+
151
134
152
-
{% endcomment %}
153
135
## Data format
154
136
155
137
Segment stores logs as gzipped, newline-separated JSON containing the full call information. For a list of supported properties, see the [Segment Spec](/docs/connections/spec/) documentation.
@@ -162,98 +144,7 @@ The received-day refers to the UTC date unix timestamp, that the API receives th
162
144
163
145
## Encryption
164
146
165
-
This section contains information for enabling encryption on your S3 bucket.
166
-
167
-
### Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)
168
-
169
-
Segment supports optional, S3-managed Server-Side Encryption, which you can disable or enable from the Destination Configuration UI. By default, the destination now automatically enables encryption, and Segment recommends that you continue to encrypt.
170
-
If you've had the S3 destination enabled since before October 2017, you might need to enable encryption manually on your bucket.
171
-
172
-
While most client libraries transparently decrypt the file when fetching it, you should make sure that any applications that are consume data in the S3 bucket are ready to decrypt the data before you enable this feature. When you're ready, you can enable encryption from the setting in the destination configuration UI.
173
-
174
-
### Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)
175
-
Segment can also write to S3 buckets with Default Encryption set to AWS-KMS. This ensures that objects written to your bucket are encrypted using customer managed keys created in your AWS Key Management Service (KMS).
176
-
Follow the steps below to enable encryption using AWS KMS Managed Keys:
177
-
178
-
#### Create a new customer-managed key and grant the Segment user permissions to generate new keys
179
-
The Segment user must have the permission to `GenerateDataKey` from your AWS Key Management Service. Here is a sample policy document that grants the Segment user the necessary permissions.
The target S3 bucket should have the "Default encryption" property enabled and set to `AWS-KMS`. Choose the customer-managed key generated in the above step for encryption.
#### Disable ServerSideEncryption in Segment S3 Destination settings
207
-
Disable the Server Side Encryption setting in the Segment destination configuration. This allows you to enable bucket-level encryption, so Amazon can encrypt objects using KMS managed keys.
To further secure your bucket by ensuring that all files upload with the encryption flag present, you can add to the bucket policy to strictly enforce that all uploads trigger encryption.
213
-
214
-
Segment recommends doing this as a best practice. The following policy strictly enforces upload encryption with Amazon S3-Managed keys.
Configure encryption at the bucket-level from within the AWS console. For more information, see Amazon's documentation [Protecting data using encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html){:target="_blank"}.
257
148
258
149
## Region
259
150
@@ -265,49 +156,13 @@ To use a custom key prefix for the files in your bucket, append the path to the
265
156
266
157
### How can I download the data from my bucket?
267
158
268
-
Segment recommends using the [AWS CLI](http://aws.amazon.com/cli/) and writing a short script to download specific days, one at a time. The AWS CLI is faster than [s3cmd](http://s3tools.org/s3cmd) because it downloads files in parallel.
269
-
270
-
> info ""
271
-
> S3 transparently decompresses the files for most clients. To access the raw gzipped data you can programmatically download the file using [the AWS SDK](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html) and setting `ResponseContentEncoding: none`. This functionality isn't available in the AWS CLI). You can also manually remove the metadata on the file (`Content-Type: text/plain` and `Content-Encoding: gzip`) through the AWS interface, which allows you to download the file as gzipped.
272
-
273
-
To configure the AWS CLI, see Amazon's documentation [here](http://docs.aws.amazon.com/cli/latest/userguide/installing.html). For linux systems, run the following command:
274
-
275
-
276
-
```bash
277
-
$ sudo apt-get install awscli
278
-
```
279
-
280
-
Then configure AWS CLI with your Access Key ID and Secret Access Key. You can create or find these keys in your [Amazon IAM user management console](https://console.aws.amazon.com/iam/home#users). Then run the following command which will prompt you for the access keys:
281
-
282
-
```bash
283
-
$ aws configure
284
-
```
285
-
286
-
To see a list of the most recent log folders:
287
-
288
-
```bash
289
-
$ aws s3 ls s3://{bucket}/segment-logs/{source-id}/ | tail -10
To put the files in a specific folder replace the `.` at the end ("current directory") with the desired directory like `~/Downloads/logs`.
159
+
Amazon provides several methods to download data from an S3 bucket. For more information, see [Downloading an object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/download-objects.html){:target="_blank"}.
305
160
306
161
307
162
## Personas
308
163
309
164
> warning ""
310
-
> As mentioned above, the Amazon S3 destination works differently than other destinations in Segment. As a result, Segment sends **all** data from a Personas source to S3 during the sync process, not only the connected audiences and traits.
165
+
> As mentioned above, the AWS S3 destination works differently than other destinations in Segment. As a result, Segment sends **all** data from a Personas source to S3 during the sync process, not only the connected audiences and traits.
311
166
312
167
You can send computed traits and audiences generated using [Segment Personas](/docs/personas) to this destination as a **user property**.
0 commit comments