-
Notifications
You must be signed in to change notification settings - Fork 393
Initial Commit #7242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
stayseesong
merged 18 commits into
segmentio:develop
from
mayur-pitale:segment-actions-s3
Dec 5, 2024
Merged
Initial Commit #7242
Changes from 16 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
4ec837f
Initial Commit
mayur-pitale 6e74d40
adding benefits and getting started
jpang007 d5d733f
adding more information + image
jpang007 3677388
missed additional line
jpang007 e1a2f9b
changing text to include benefit over classic
jpang007 aad45ce
Changed descriptions
mayur-pitale 2eabdf1
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale 28bc02c
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale be1df3a
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale 8d27dc5
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale d10cff7
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale f7d36e6
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale 75869ae
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale 175054b
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale fb7a161
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale 680b42a
Update src/connections/destinations/catalog/actions-s3/index.md
mayur-pitale 048a0d8
Removed Actions section
mayur-pitale a0d60be
Update index.md
stayseesong File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
102 changes: 102 additions & 0 deletions
102
src/connections/destinations/catalog/actions-s3/index.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,102 @@ | ||
--- | ||
title: AWS S3 (Actions) Destination | ||
hide-boilerplate: true | ||
hide-dossier: false | ||
id: 66eaa166f650644f04389e2c | ||
# versions: | ||
# - name: AWS S3 (Classic) | ||
# link: /docs/connections/destinations/catalog/aws-s3/ | ||
--- | ||
{% include content/plan-grid.md name="actions" %} | ||
|
||
The AWS S3 (Actions) destination allows you to store event data as objects in a secure, scalable cloud storage solution. Each event is written to your S3 bucket, organized into a customizable folder structure such as by event type or timestamp. This makes it easy to manage, archive, and analyze data using downstream tools or AWS services. | ||
|
||
|
||
## Benefits of AWS S3 (Actions) vs AWS S3 Classic | ||
The traditional AWS S3 Classic destination enabled the storage of raw logs containing data Segment received, directly into your S3 bucket. While this provided a straightforward data storage solution, users often needed to implement additional processing to standardize or transform these logs (in JSON format) for downstream analytics or integrations. | ||
|
||
The AWS S3 (Actions) destination enhances this capability by introducing configurable options to format and structure event data prior to storage. This new approach offers several key benefits: | ||
|
||
* **Standardized Data Formatting**. AWS S3 (Actions) lets you define consistent output formats for your data, either CSV or TXT file formats, in a folder definition that you choose. The previous AWS S3 Classic Destination only allowed raw JSON payloads stored within a specific folder called `"segment-logs"`. | ||
|
||
* **Configurable Data Translation**. AWS S3 (Actions) supports translation rules that can map raw event attributes to more meaningful or actionable representations. You can configure these rules to meet specific data schema requirements by either adding in custom columns or using the default ones. | ||
|
||
* **Enhanced Delivery Controls**. The destination provides advanced options for batch size controls and file naming conventions. These controls can help optimize efficiency and simplify data retrieval workflows. | ||
|
||
## Getting started | ||
Setting up the AWS S3 (Actions) destination is a straightforward process designed to help you configure and deploy standardized event data to your Amazon S3 bucket. Follow these steps to get started: | ||
|
||
### Prerequisites | ||
Ensure you have the following in place before configuring the AWS S3 (Actions) destination: | ||
|
||
- Amazon S3 Bucket: Create a bucket in your AWS account or use an existing one where you want to store the event data. | ||
- AWS IAM Permissions: Verify that you have appropriate IAM roles with write access to the S3 bucket and permissions for the Segment connection. | ||
- IAM Access IDs: Prepare your AWS IAM ARN ID and IAM External ID. These will be needed to authenticate and authorize Segment with your S3 bucket. | ||
|
||
|
||
### Step 1: Create an IAM role in the AWS console | ||
To set up the IAM role to properly authorize Segment with the AWS S3 (Actions) destination: | ||
|
||
1. Log in to your AWS account. | ||
2. Create a new or use an existing bucket with `PutObject`, `GetObject`, `ListObject` access to the S3 bucket. | ||
3. Navigate to **IAM > Roles > Create Role**. | ||
4. Provide the following policy permissions for the IAM that was just created: | ||
```json | ||
{ | ||
"Version": "2012-10-17", | ||
"Statement": [ | ||
{ | ||
"Sid": "PutObjectsInBucket", | ||
"Effect": "Allow", | ||
"Action": [ | ||
"s3:PutObject", | ||
"s3:PutObjectAcl" | ||
], | ||
"Resource": "arn:aws:s3:::<YOUR_BUCKET_NAME>/*" | ||
} | ||
] | ||
} | ||
``` | ||
5. Click on the Trust Relationships tab and edit the trust policy to allow the IAM user to assume the role. If a user is not already created, refer to the AWS documentation to create a user. | ||
```json | ||
{ | ||
"Version": "2012-10-17", | ||
"Statement": [ | ||
{ | ||
"Sid": "", | ||
"Effect": "Allow", | ||
"Principal": { | ||
"AWS": | ||
"arn:aws:iam::595280932656:role/customer-s3-prod-action-destination-access" | ||
}, | ||
"Action": "sts:AssumeRole", | ||
"Condition": { | ||
"StringEquals": { | ||
"sts:ExternalId": "<YOUR_EXTERNAL_ID>" | ||
} | ||
} | ||
} | ||
] | ||
} | ||
``` | ||
## Step 2: Add the AWS S3 (Actions) Destination in Segment | ||
To finish the configuration, enable the AWS S3 (Actions) Destination in your workspace. | ||
|
||
1. Add the **AWS S3 (Actions)** destination from the Destinations tab of the catalog. | ||
2. Select the data source you want to connect to the destination. | ||
3. Provide a unique name for the destination. | ||
4. Complete the destination settings: | ||
* Enter the name of the region in which the bucket you created above resides. | ||
* Enter the name of the bucket you created above. Be sure to enter the bucket's **name** and not URI. | ||
* Enter the ARN of the IAM role you created above. The ARN should follow the format `arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME.` | ||
* Enter the IAM External ID, which is a value set in the Trust Relationship under your AWS IAM Role. | ||
5. Enable the destination. | ||
6. Verify the Segment data is stored in the S3 bucket by navigating to the `<your_S3_bucket>/` in the AWS console. | ||
|
||
### Actions | ||
Build your own Mappings. The following action is supported: | ||
- [Sync to S3](#sync-to-s3) | ||
|
||
{% include components/actions-fields.html settings="true"%} | ||
|
||
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mayur-pitale sorry, one last thing, where is this linking to? there is no #syn-to-s3 section on this page