You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/connections/storage/catalog/aws-s3/index.md
+73-7Lines changed: 73 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -212,7 +212,7 @@ To finish configuration, enable the AWS S3 Destination with IAM Role Support des
212
212
> Did you know you can create destinations with the Config API? For more information, see [Create Destination](https://reference.segmentapis.com/#51d965d3-4a67-4542-ae2c-eb1fdddc3df6){:target="_blank"}.
213
213
214
214
215
-
## Migrate an existing destination
215
+
## Manually migrate an existing destination
216
216
217
217
> warning "Avoid overwriting data"
218
218
> Sending data to the same S3 location from both the existing Amazon S3 destination, and the AWS S3 with IAM Role Support destination will overwrite data in that location. To avoid this, follow the steps below.
@@ -227,7 +227,7 @@ To migrate an existing Amazon S3 destination to the AWS S3 with IAM Role Support
227
227
6. Remove the test folder created in step 2 from the bucket.
228
228
229
229
> error " "
230
-
> You need to migrate to the new S3 destination before you disable your legacy destination to ensure Segment continues to deliver data to your S3 bucket.
230
+
> You must migrate to the new S3 destination before you disable your legacy destination to ensure Segment continues to deliver data to your S3 bucket.
231
231
232
232
### Migration steps for scenarios with multiple sources per environment
233
233
@@ -244,10 +244,76 @@ For example:
244
244
245
245
For each source in the scenario, complete the steps described in [Migrate an existing destination](#migrate-an-existing-destination), and ensure that you have separate IAM Roles and Permissions set for staging and production use.
246
246
247
-
### Test your migrated source
247
+
248
+
## Migrate an existing destination using public-api
249
+
This procedure uses Segment's public-api to migrate your destinations from Amazon S3 to the AWS S3 destination. For more information about the public-api, please see the [Getting Started](https://api.segmentapis.com/docs/guides/#getting-started) guide.
250
+
251
+
To migrate from the Amazon S3 destination to the AWS S3 destination using the public-api:
252
+
253
+
1. Add the **AWS S3** destination and add the AWS Region and IAM role ARN. For the bucket name, enter `<YOUR_BUCKET_NAME>/segment-logs/test`. Enable the destination, and verify data is received at `<YOUR_BUCKET_NAME>/segment-logs/test/segment-logs`. If the folder receives data, continue to the next step. If you don't see log entries, check the trust relationship document and IAM policy attached to the role.
254
+
2. Get a list of destinations or an individual destination. To return a list of all of your Amazon S3 destinations, use the `list desintations` call and filter the results using metadata id `54f418c3db31d978f14aa925` or slug `amazon-s3`: <br/>
255
+
```shell
256
+
curl -vvv --location --request GET https://api.segmentapis.com/destinations?pagination.count=1 \
257
+
--header 'Content-Type: application/json' \
258
+
--header 'Authorization: Bearer ...' \
259
+
--data-raw '
260
+
```
261
+
<br/> To return the information for an individual Amazon S3 destination, use the `get destination` call, using the destination ID for your individual Amazon S3 destination (**Note:** Destination IDs are visible in the Segment app, in the settings for the source.) <br/>
262
+
```shell
263
+
curl -vvv --location --request GET https://api.segmentapis.com/destinations/$DESTINATION_ID \
264
+
--header 'Content-Type: application/json' \
265
+
--header 'Authorization: Bearer ...' \
266
+
--data-raw '
267
+
```
268
+
269
+
3. Create the new AWS S3 destination using the `create destination` call. Below is an example of the parameters: <br/>
<br/>**Optional:** You can create a destination that is not enabled automatically upon creation by setting `enabled` to `false` when creating the new AWS S3 destination:
4. After creating your new AWS S3 destination, delete the Amazon S3 sources using the following command (replacing `$DESTINATION_ID` with the ID of your Amazon S3 destination):
> Sending data to the same S3 location from both the existing Amazon S3 destination and the AWS S3 destinations will overwrite data in your instance of S3. To avoid this, disable your Amazon S3 destination after you create your AWS S3 destination.
312
+
313
+
## Test your migrated source
248
314
You can validate that your configured your migrated source correctly in the Settings section of the AWS S3 destination.
249
315
250
-
> important "Source editing permissions required"
316
+
> success "Source editing permissions required"
251
317
> In-app source validation is restricted to users with source editing permissions (for example, users with Workspace Owner, Source Admin, or Workspace Admin roles). For more information about roles in the Segment app, please see the [Roles documentation](/docs/segment-app/iam/roles/).
252
318
253
319
To verify that you migrated your source correctly:
@@ -259,18 +325,18 @@ To verify that you migrated your source correctly:
259
325
> note "`dummy-object.txt`"
260
326
> In order to test your bucket, Segment will upload a text file, `dummy-object.txt`, to your `segment-logs` folder. After you've completed the validation process, feel free to delete this file.
261
327
262
-
#### Troubleshooting
328
+
### Troubleshooting
263
329
264
330
The following table outlines some of the error codes the validation tool may display and possible reasons for the error.
265
331
266
332
| Error message | Likely cause of the error |
267
333
| ------------- | ------------------------- |
268
-
| Unknown Error. Please try again. If the problem persists, please contact [Segment support](mailto:[email protected]) | Fail to assume intermediate role |
334
+
| Unknown Error. Please try again. If the problem persists, please contact [Segment support](mailto:[email protected]).| Fail to assume intermediate role |
269
335
| Access Denied. Please configure External ID in the AWS IAM Console. [Learn more](#create-an-iam-role-in-the-aws-console). | Successfully assumed customer's role, role doesn't have external ID |
270
336
| Unknown Error. Please follow [instructions](#create-an-iam-role-in-the-aws-console) to set up the AWS S3 destination. If the problem persists, please contact [Segment support](mailto:[email protected]). | Fail to assume customer’s role without external ID & returned an error code that is not error 403 |
271
337
| Access Denied. Please configure External ID in the AWS IAM Console. [Learn more](#create-an-iam-role-in-the-aws-console). | Fail to assume customer’s role without external ID & returned error 403 |
272
338
| Access Denied. Please add PutObject permissions to the IAM role in the AWS IAM Console. [Learn more](#create-an-iam-role-in-the-aws-console). | Fail to upload the dummy object to customer's S3 bucket & an error code that is not error 403. |
273
-
| Unknown Error. Please follow [instructions](#create-an-iam-role-in-the-aws-console) to set up the AWS s3 destination. If the problem persists, please contact [Segment support](mailto:[email protected]). | Fail to upload the dummy object to customer's S3 bucket & an error code that is not error 403. |
339
+
| Unknown Error. Please follow [instructions](#create-an-iam-role-in-the-aws-console) to set up the AWS S3 destination. If the problem persists, please contact [Segment support](mailto:[email protected]). | Fail to upload the dummy object to customer's S3 bucket & an error code that is not error 403. |
0 commit comments