You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/connections/destinations/catalog/amazon-kinesis-firehose/index.md
+49-45Lines changed: 49 additions & 45 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,22 +2,23 @@
2
2
rewrite: true
3
3
title: Amazon Kinesis Firehose Destination
4
4
---
5
-
[Amazon Kinesis Firehose](https://aws.amazon.com/kinesis/data-firehose/)is the easiest way to load streaming data into AWS. It can capture, transform, and load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards you're already using today. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress, and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security.
5
+
[Amazon Kinesis Firehose](https://aws.amazon.com/kinesis/data-firehose/)provides way to load streaming data into AWS. It can capture, transform, and load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards you're already using today. It's a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress, and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security.
6
6
7
7
This document was last updated on February 05, 2020. If you notice any gaps, outdated information or simply want to leave some feedback to help us improve our documentation, [let us know](https://segment.com/help/contact)!
8
8
9
9
## Getting Started
10
10
11
11
{% include content/connection-modes.md %}
12
12
13
-
1. Create at least one Kinesis Firehose delivery stream. You can follow these [instructions](http://docs.aws.amazon.com/firehose/latest/dev/basic-create.html) to create a new delivery stream.
13
+
To get started:
14
+
1. Create at least one Kinesis Firehose delivery stream. You can follow these [instructions](http://docs.aws.amazon.com/firehose/latest/dev/basic-create.html){:target="_blank"} to create a new delivery stream.
14
15
2. Create an IAM policy.
15
-
1. Sign in to the [Identity and Access Management (IAM) console](https://console.aws.amazon.com/iam/).
16
-
2. Follow [these instructions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html#access_policies_create-json-editor) to create an IAM policy on the JSON to allow Segment permission to write to your Kinesis Firehose Stream.
16
+
1. Sign in to the [Identity and Access Management (IAM) console](https://console.aws.amazon.com/iam/){:target="_blank"}.
17
+
2. Follow [these instructions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html#access_policies_create-json-editor){:target="_blank"} to create an IAM policy on the JSON to allow Segment permission to write to your Kinesis Firehose Stream.
17
18
- Use the following template policy in the **Policy Document** field. Be sure to change the `{region}`, `{account-id}` and `{stream-name}` with the applicable values.
To begin using the Kinesis Firehose destination, you must first decide on which Segment events you would like to route to which Firehose delivery streams. This mapping then needs to be defined in your destination settings.
77
78
78
-
Segment `track` events can map based on their **event name**. For example, if you have an event called `User Registered`, and you want these events to be published to a Firehose delivery stream called `new_users`, you would create a row in your destination settings that looks like this:
79
+
Segment `track` events can map based on their **event name**. For example, if you have an event called `User Registered`, and you want these events to be published to a Firehose delivery stream called `new_users`, create a row in your destination settings that looks like this:
Any Segment **event type** (ie. `page`, `track`, `identify`, `screen`, etc.) can also be mapped. This enables you to publish all instances of a given Segment event type to a given stream. To do this, create a row with the event type and its corresponding delivery stream:
Events can be defined **insensitive to case** so `Page` will be equivalent to `page`. The delivery stream name however needs to be formatted exactly as it is on AWS.
87
+
Events can be defined **insensitive to case** so `Page` will be equivalent to `page`. The delivery stream name needs to be formatted exactly as it is on AWS.
87
88
88
89
If you would like to route all events to a stream, use an `*` as the event name.
89
90
@@ -103,7 +104,7 @@ Let's say you've decided to publish your Segment track events named `User Regist
103
104
104
105
The Segment Kinesis destination will issue a `PutRecord` request with the following parameters:
105
106
106
-
```
107
+
```js
107
108
firehose.putRecord({
108
109
Record: {
109
110
Data:JSON.stringify(msg)) +'/n'
@@ -112,7 +113,7 @@ firehose.putRecord({
112
113
});
113
114
```
114
115
115
-
Segment will append a newline character to each record to allow for easy downstream parsing.
116
+
Segment appends a newline character to each record to allow for easy downstream parsing.
116
117
117
118
## Group
118
119
Take a look to understand what the [Group method](https://segment.com/docs/connections/spec/group/) does. An example group call is shown below:
If you have multiple sources using Kinesis/Firehose, you have two options:
133
134
134
135
#### Attach multiple sources to your IAM role
135
-
Find the IAM role you created for this destination in the AWS Console in Services > IAM > Roles. Click on the role, and navigate to the **Trust Relationships** tab. Click **Edit trust relationship**. You should see a snippet that looks something that looks like this:
136
+
To attach multiple sources to your IAM role:
137
+
1. Find the IAM role you created for this destination in the AWS Console in **Services > IAM > Roles**.
138
+
2. Select the role and navigate to the **Trust Relationships** tab.
139
+
3. Click **Edit trust relationship**. You should see a snippet that looks something that looks like this:
136
140
137
-
```json
138
-
{
139
-
"Version": "2012-10-17",
140
-
"Statement": [
141
+
```json
141
142
{
142
-
"Effect": "Allow",
143
-
"Principal": {
144
-
"AWS": "arn:aws:iam::595280932656:root"
145
-
},
146
-
"Action": "sts:AssumeRole",
147
-
"Condition": {
148
-
"StringEquals": {
149
-
"sts:ExternalId": "YOUR_SEGMENT_SOURCE_ID"
143
+
"Version": "2012-10-17",
144
+
"Statement": [
145
+
{
146
+
"Effect": "Allow",
147
+
"Principal": {
148
+
"AWS": "arn:aws:iam::595280932656:root"
149
+
},
150
+
"Action": "sts:AssumeRole",
151
+
"Condition": {
152
+
"StringEquals": {
153
+
"sts:ExternalId": "YOUR_SEGMENT_SOURCE_ID"
154
+
}
155
+
}
150
156
}
151
-
}
157
+
]
152
158
}
153
-
]
154
-
}
155
-
```
159
+
```
156
160
157
-
Replace that snippet with the following, and replace the contents of the array with all of your source IDs.
161
+
4. Replace that snippet with the following, and replace the contents of the array with all of your source IDs.
If you have many sources using Kinesis that it's impractical to attach all of their IDs to your IAM role, you can set a single ID to use instead. *This approach requires that you securely store a secret value, so we recommend that you use the method above if at all possible.*
185
+
If you have many sources using Kinesis that it's impractical to attach all of their IDs to your IAM role, you can set a single ID to use instead.
182
186
183
187
To set this value for a single Secret ID:
184
188
1. Go to the Kinesis Firehose destination settings from each of your Segment sources.
0 commit comments