Skip to content

Commit 26020e3

Browse files
committed
Adding first pass of the new migration procedure
1 parent 5adeed9 commit 26020e3

File tree

1 file changed

+73
-7
lines changed
  • src/connections/storage/catalog/aws-s3

1 file changed

+73
-7
lines changed

src/connections/storage/catalog/aws-s3/index.md

Lines changed: 73 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,7 @@ To finish configuration, enable the AWS S3 Destination with IAM Role Support des
212212
> Did you know you can create destinations with the Config API? For more information, see [Create Destination](https://reference.segmentapis.com/#51d965d3-4a67-4542-ae2c-eb1fdddc3df6){:target="_blank"}.
213213

214214

215-
## Migrate an existing destination
215+
## Manually migrate an existing destination
216216

217217
> warning "Avoid overwriting data"
218218
> Sending data to the same S3 location from both the existing Amazon S3 destination, and the AWS S3 with IAM Role Support destination will overwrite data in that location. To avoid this, follow the steps below.
@@ -227,7 +227,7 @@ To migrate an existing Amazon S3 destination to the AWS S3 with IAM Role Support
227227
6. Remove the test folder created in step 2 from the bucket.
228228

229229
> error " "
230-
> You need to migrate to the new S3 destination before you disable your legacy destination to ensure Segment continues to deliver data to your S3 bucket.
230+
> You must migrate to the new S3 destination before you disable your legacy destination to ensure Segment continues to deliver data to your S3 bucket.
231231

232232
### Migration steps for scenarios with multiple sources per environment
233233

@@ -244,10 +244,76 @@ For example:
244244

245245
For each source in the scenario, complete the steps described in [Migrate an existing destination](#migrate-an-existing-destination), and ensure that you have separate IAM Roles and Permissions set for staging and production use.
246246

247-
### Test your migrated source
247+
248+
## Migrate an existing destination using public-api
249+
This procedure uses Segment's public-api to migrate your destinations from Amazon S3 to the AWS S3 destination. For more information about the public-api, please see the [Getting Started](https://api.segmentapis.com/docs/guides/#getting-started) guide.
250+
251+
To migrate from the Amazon S3 destination to the AWS S3 destination using the public-api:
252+
253+
1. Add the **AWS S3** destination and add the AWS Region and IAM role ARN. For the bucket name, enter `<YOUR_BUCKET_NAME>/segment-logs/test`. Enable the destination, and verify data is received at `<YOUR_BUCKET_NAME>/segment-logs/test/segment-logs`. If the folder receives data, continue to the next step. If you don't see log entries, check the trust relationship document and IAM policy attached to the role.
254+
2. Get a list of destinations or an individual destination. To return a list of all of your Amazon S3 destinations, use the `list desintations` call and filter the results using metadata id `54f418c3db31d978f14aa925` or slug `amazon-s3`: <br/>
255+
```shell
256+
curl -vvv --location --request GET https://api.segmentapis.com/destinations?pagination.count=1 \
257+
--header 'Content-Type: application/json' \
258+
--header 'Authorization: Bearer ...' \
259+
--data-raw '
260+
```
261+
<br/> To return the information for an individual Amazon S3 destination, use the `get destination` call, using the destination ID for your individual Amazon S3 destination (**Note:** Destination IDs are visible in the Segment app, in the settings for the source.) <br/>
262+
```shell
263+
curl -vvv --location --request GET https://api.segmentapis.com/destinations/$DESTINATION_ID \
264+
--header 'Content-Type: application/json' \
265+
--header 'Authorization: Bearer ...' \
266+
--data-raw '
267+
```
268+
269+
3. Create the new AWS S3 destination using the `create destination` call. Below is an example of the parameters: <br/>
270+
```json
271+
{
272+
"sourceId": "rh5BDZp6QDHvXFCkibm1pR",
273+
"metadataId": "60be92c8dabdd561bf6c9130",
274+
"name": "AWS S3",
275+
"settings": {
276+
"region": "XYZ",
277+
"s3Bucket": "test",
278+
"iamRoleArn": "arn:aws:iam::355207333203:role/organization-s3-copy-secure"
279+
}
280+
}
281+
```
282+
<br/>**Optional:** You can create a destination that is not enabled automatically upon creation by setting `enabled` to `false` when creating the new AWS S3 destination:
283+
<br/>
284+
```shell
285+
curl -vvv --location --request PATCH https://api.segmentapis.com/destinations/$DESTINATION_ID \
286+
--header 'Content-Type: application/json' \
287+
--header 'Authorization: Bearer ...' \
288+
--data-raw '
289+
{
290+
"destinationId": "$DESTINATION_ID",
291+
"enabled": true
292+
}
293+
' | jq
294+
```
295+
<br/>
296+
4. After creating your new AWS S3 destination, delete the Amazon S3 sources using the following command (replacing `$DESTINATION_ID` with the ID of your Amazon S3 destination):
297+
298+
```shell
299+
curl -vvv --location --request PATCH https://api.segmentapis.com/destinations/$DESTINATION_ID \
300+
--header 'Content-Type: application/json' \
301+
--header 'Authorization: Bearer ...' \
302+
--data-raw '
303+
{
304+
"destinationId": "$DESTINATION_ID",
305+
"enabled": false
306+
}
307+
' | jq
308+
```
309+
310+
> warning "Avoid overwriting data"
311+
> Sending data to the same S3 location from both the existing Amazon S3 destination and the AWS S3 destinations will overwrite data in your instance of S3. To avoid this, disable your Amazon S3 destination after you create your AWS S3 destination.
312+
313+
## Test your migrated source
248314
You can validate that your configured your migrated source correctly in the Settings section of the AWS S3 destination.
249315
250-
> important "Source editing permissions required"
316+
> success "Source editing permissions required"
251317
> In-app source validation is restricted to users with source editing permissions (for example, users with Workspace Owner, Source Admin, or Workspace Admin roles). For more information about roles in the Segment app, please see the [Roles documentation](/docs/segment-app/iam/roles/).
252318
253319
To verify that you migrated your source correctly:
@@ -259,18 +325,18 @@ To verify that you migrated your source correctly:
259325
> note "`dummy-object.txt`"
260326
> In order to test your bucket, Segment will upload a text file, `dummy-object.txt`, to your `segment-logs` folder. After you've completed the validation process, feel free to delete this file.
261327

262-
#### Troubleshooting
328+
### Troubleshooting
263329

264330
The following table outlines some of the error codes the validation tool may display and possible reasons for the error.
265331

266332
| Error message | Likely cause of the error |
267333
| ------------- | ------------------------- |
268-
| Unknown Error. Please try again. If the problem persists, please contact [Segment support](mailto:[email protected]) | Fail to assume intermediate role |
334+
| Unknown Error. Please try again. If the problem persists, please contact [Segment support](mailto:[email protected]). | Fail to assume intermediate role |
269335
| Access Denied. Please configure External ID in the AWS IAM Console. [Learn more](#create-an-iam-role-in-the-aws-console). | Successfully assumed customer's role, role doesn't have external ID |
270336
| Unknown Error. Please follow [instructions](#create-an-iam-role-in-the-aws-console) to set up the AWS S3 destination. If the problem persists, please contact [Segment support](mailto:[email protected]). | Fail to assume customer’s role without external ID & returned an error code that is not error 403 |
271337
| Access Denied. Please configure External ID in the AWS IAM Console. [Learn more](#create-an-iam-role-in-the-aws-console). | Fail to assume customer’s role without external ID & returned error 403 |
272338
| Access Denied. Please add PutObject permissions to the IAM role in the AWS IAM Console. [Learn more](#create-an-iam-role-in-the-aws-console). | Fail to upload the dummy object to customer's S3 bucket & an error code that is not error 403. |
273-
| Unknown Error. Please follow [instructions](#create-an-iam-role-in-the-aws-console) to set up the AWS s3 destination. If the problem persists, please contact [Segment support](mailto:[email protected]). | Fail to upload the dummy object to customer's S3 bucket & an error code that is not error 403. |
339+
| Unknown Error. Please follow [instructions](#create-an-iam-role-in-the-aws-console) to set up the AWS S3 destination. If the problem persists, please contact [Segment support](mailto:[email protected]). | Fail to upload the dummy object to customer's S3 bucket & an error code that is not error 403. |
274340

275341

276342
## Data format

0 commit comments

Comments
 (0)