You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Segment Data Lakes provide a way to collect large quantities of data in a format that's optimized for targeted data science and data analytics workflows. You can read [more information about Data Lakes](/docs/connections/storage/data-lakes/) and learn [how they differ from warehouses](/docs/connections/storage/data-lakes/comparison/) in our documentation.
6
+
Segment Data Lakes provide a way to collect large quantities of data in a format that's optimized for targeted data science and data analytics workflows. You can read [more information about Data Lakes](/docs/connections/storage/data-lakes/) and learn [how they differ from Warehouses](/docs/connections/storage/data-lakes/comparison/) in our documentation.
8
7
9
8
> info ""
10
9
> Segment Data Lakes is available to Business tier customers only.
@@ -13,15 +12,15 @@ Segment Data Lakes provide a way to collect large quantities of data in a format
13
12
14
13
Before you set up Segment Data Lakes, you need the following resources:
15
14
16
-
- An authorized [AWS account](https://aws.amazon.com/account/)
17
-
- An [Amazon S3 bucket](https://github.com/terraform-aws-modules/terraform-aws-s3-bucket) to send data to and store logs
15
+
- An [AWS account](https://aws.amazon.com/account/)
16
+
- An [Amazon S3 bucket](https://github.com/terraform-aws-modules/terraform-aws-s3-bucket) to receive data and store logs
18
17
- A subnet within a VPC for the EMR cluster to run in
19
18
20
19
## Step 1 - Set Up AWS Resources
21
20
22
-
You can use the [open source Terraform module](https://github.com/segmentio/terraform-aws-data-lake) to automate much of the set up work to get Data Lakes up and running. If you’re familiar with Terraform, you can modify the module to meet your organization’s needs, however we can only guarantee support for the template as provided. The Terraform version should be > 0.12.
21
+
You can use the [open source Terraform module](https://github.com/segmentio/terraform-aws-data-lake) to automate much of the set up work to get Data Lakes up and running. If you’re familiar with Terraform, you can modify the module to meet your organization’s needs, however Segment guarantees support only for the template as provided. The Data Lakes set up uses Terraform v0.11+. To support more versions of Terraform, the aws provider must use v2, which is included in our example main.tf.
23
22
24
-
You can also use our [manual set up instructions](https://docs.google.com/document/d/1GlWzS5KO4QaiVZx9pwfpgF-N-Xy2e_QQcdYSX-nLMDU/view) to configure these AWS resources if you prefer.
23
+
You can also use our [manual set up instructions](/docs/connections/storage/data-lakes/data-lakes-manual-setup) to configure these AWS resources if you prefer.
25
24
26
25
The Terraform module and manual set up instructions both provide a base level of permissions to Segment (for example, the correct IAM role to allow Segment to create Glue databases on your behalf). If you want stricter permissions, or other custom configurations, you can customize these manually.
27
26
@@ -31,53 +30,61 @@ After you set up the necessary AWS resources, the next step is to set up the Dat
31
30
32
31
1. In the [Segment App](https://app.segment.com/goto-my-workspace/overview), click **Add Destination**, then search for and select **Data Lakes**.
33
32
34
-
2. Click **Configure Data Lakes** and select the source to connect to the Data Lakes destination.
35
-
> **Warning**:You must include all source ids in the external ID list in the IAM policy, or else the source data cannot be synced to S3.
33
+
2. Click **Configure Data Lakes** and select the source to connect to the Data Lakes destination.
34
+
**Warning**:You must include all source ids in the external ID list in the IAM policy, or else the source data cannot be synced to S3.
36
35
37
-
4. In the Settings tab, enter and save the following connection settings:
38
-
-**AWS Region**: The AWS Region where your EMR cluster, S3 Bucket and Glue DB reside.
36
+
3. In the Settings tab, enter and save the following connection settings:
37
+
-**AWS Region**: The AWS Region where your EMR cluster, S3 Bucket and Glue DB reside. Ex: `us-west-2`
39
38
-**EMR Cluster ID**: The EMR Cluster ID where the Data Lakes jobs will be run.
40
39
-**Glue Catalog ID**: The Glue Catalog ID (this must be the same as your AWS account ID).
41
-
-**IAM Role ARN**: The ARN of the IAM role that Segment will use to connect to Data Lakes.
42
-
-**S3 Bucket**: Name of the S3 bucket used by Data Lakes. The EMR cluster will store logs in this bucket.
43
-
40
+
-**IAM Role ARN**: The ARN of the IAM role that Segment will use to connect to Data Lakes. Ex: `arn:aws:iam::000000000000:role/SegmentDataLakeRole`
41
+
-**S3 Bucket**: Name of the S3 bucket used by Data Lakes. The EMR cluster will store logs in this bucket. Ex: `segment-data-lake`
42
+
44
43
You must individually connect each source to the Data Lakes destination. However, you can copy the settings from another source by clicking **…** ("more") (next to the button for “Set up Guide”).
45
44
46
-
5._(Optional)_**Date Partition**: Optional advanced setting to change the date partition structure, with a default structure `day=<YYYY-MM-DD>/hr=<HH>`. To use the default, leave this setting unchanged. To partition the data by a different date structure, choose one of the following options:
45
+
4._(Optional)_**Date Partition**: Optional advanced setting to change the date partition structure, with a default structure `day=<YYYY-MM-DD>/hr=<HH>`. To use the default, leave this setting unchanged. To partition the data by a different date structure, choose one of the following options:
47
46
- Day/Hour [YYYY-MM-DD/HH] (Default)
48
47
- Year/Month/Day/Hour [YYYY/MM/DD/HH]
49
48
- Year/Month/Day [YYYY/MM/DD]
50
49
- Day [YYYY-MM-DD]
51
50
52
-
6._(Optional)_**Glue Database Name**: Optional advanced setting to change the name of the Glue Database which is set to the source slug by default. Each source connected to Data Lakes must have a different Glue Database name otherwise data from different sources will collide in the same database.
51
+
5._(Optional)_**Glue Database Name**: Optional advanced setting to change the name of the Glue Database which is set to the source slug by default. Each source connected to Data Lakes must have a different Glue Database name otherwise data from different sources will collide in the same database.
53
52
54
-
7. Enable the Data Lakes destination by clicking the toggle near the **Set up Guide** button.
53
+
6. Enable the Data Lakes destination by clicking the toggle near the **Set up Guide** button.
55
54
56
55
Once the Data Lakes destination is enabled, the first sync will begin approximately 2 hours later.
57
56
58
57
59
-
## (Optional) Step 3 - Replay Historical Data
58
+
## Step 3 - Verify Data is Synced to S3 and Glue
60
59
61
-
If you want to add historical data to your data set using a [replay of historical data](/docs/guides/what-is-replay/) into Data Lakes, [contact the Segment Support team](https://segment.com/help/contact/) to request one.
60
+
You will see event data and [sync reports](/docs/connections/storage/data-lakes/sync-reports) populated in S3 and Glue after the first sync successfully completes. However if an [insufficient permission](/docs/connections/storage/data-lakes/sync-reports/#insufficient-permissions) or [invalid setting](/docs/connections/storage/data-lakes/sync-reports/#invalid-settings) is provided during set up, the first data lake sync will fail.
62
61
63
-
The time needed to process a Replay can vary depending on the volume of data and number of events in each source. If you decide to run a Replay, we recommend that you start with data from the last six months to get started, and then replay additional data if you find you need more.
62
+
To be alerted of sync failures via email, subscribe to the `Storage Destination Sync Failed` activity email notification within the App Settings > User Preferences > [Notification Settings](https://app.segment.com/goto-my-workspace/settings/notifications).
63
+
64
64
65
-
Segment uses a creates a separate EMR cluster to run replays, then destroys it when the replay finished. This ensures that regular Data Lakes syncs are not interrupted, and helps the replay finish faster.
65
+
`Sync Failed` emails are sent on the 1st, 5th and 20th sync failure. Learn more about the types of errors which can cause sync failures [here](/docs/connections/storage/data-lakes/sync-reports/#sync-errors).
66
66
67
-
# Common Questions
68
67
69
-
## Data Lakes Set Up
68
+
## (Optional) Step 4 - Replay Historical Data
70
69
71
-
##### Do I need to create Glue databases?
70
+
If you want to add historical data to your data set using a [replay of historical data](/docs/guides/what-is-replay/) into Data Lakes, [contact the Segment Support team](https://segment.com/help/contact/) to request one.
72
71
73
-
No, Data Lakes automatically creates one Glue database per source. This database uses the source slug as its name.
72
+
The time needed to process a Replay can vary depending on the volume of data and number of events in each source. If you decide to run a Replay, we recommend that you start with data from the last six months to get started, and then replay additional data if you find you need more.
74
73
75
-
##### What IAM role do I use in the Settings page?
74
+
Segment creates a separate EMR cluster to run replays, then destroys it when the replay finished. This ensures that regular Data Lakes syncs are not interrupted, and helps the replay finish faster.
76
75
77
-
Four roles are created when you set up Data Lakes using Terraform. You add the `arn:aws:iam::$ACCOUNT_ID:role/segment-data-lake-iam-role` role to the Data Lakes Settings page in the Segment web app.
76
+
## FAQ
78
77
79
-
##### What level of access do the AWS roles have?
78
+
###Data Lakes Set Up
80
79
80
+
{% faq %}
81
+
{% faqitem Do I need to create Glue databases? %}
82
+
No, Data Lakes automatically creates one Glue database per source. This database uses the source slug as its name.
83
+
{% endfaqitem %}
84
+
{% faqitem What IAM role do I use in the Settings page? %}
85
+
Four roles are created when you set up Data Lakes using Terraform. You add the `arn:aws:iam::$ACCOUNT_ID:role/segment-data-lake-iam-role` role to the Data Lakes Settings page in the Segment web app.
86
+
{% endfaqitem %}
87
+
{% faqitem What level of access do the AWS roles have? %}
81
88
The roles which Data Lakes assigns during set up are:
82
89
83
90
-**`segment-datalake-iam-role`** - This is the role that Segment assumes to access S3, Glue and the EMR cluster. It allows Segment access to:
@@ -92,61 +99,49 @@ The roles which Data Lakes assigns during set up are:
92
99
- Access only to the specific S3 bucket used for Data Lakes.
93
100
94
101
-**`segment_emr_autoscaling_role`** - Restricted role that can only be assumed by EMR and EC2. This is set up based on [AWS best practices](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-role-automatic-scaling.html).
95
-
96
-
##### Why doesn't the Data Lakes Terraform module create an S3 bucket?
97
-
102
+
{% endfaqitem %}
103
+
{% faqitem Why doesn't the Data Lakes Terraform module create an S3 bucket? %}
98
104
The module doesn't create a new S3 bucket so you can re-use an existing bucket for your Data Lakes.
99
-
100
-
##### Does my S3 bucket need to be in the same region as the other infrastructure?
101
-
105
+
{% endfaqitem %}
106
+
{% faqitem Does my S3 bucket need to be in the same region as the other infrastructure? %}
102
107
Yes, the S3 bucket and the EMR cluster must be in the same region.
103
-
104
-
##### How do I connect a new source to Data Lakes?
105
-
108
+
{% endfaqitem %}
109
+
{% faqitem How do I connect a new source to Data Lakes? %}
106
110
To connect a new source to Data Lakes:
107
111
108
112
1. Add the `source_id` found in the Segment workspace into the list of [external ids](https://github.com/segmentio/terraform-aws-data-lake/tree/master/modules/iam#external_ids) in the IAM policy. You can either update this from the AWS console, or re-run the [Terraform](https://github.com/segmentio/terraform-aws-data-lake) job.
109
113
2. From your Segment workspace, connect the source to the Data Lakes destination.
110
-
111
-
##### Can I configure multiple sources to use the same EMR cluster?
112
-
113
-
Yes, you can configure multiple sources to use the same EMR cluster. We recommend that the EMR cluster only be used for Data Lakes to ensure there aren't interruptions from non-Data Lakes jobs.
114
-
115
-
116
-
## Post-Set Up
117
-
118
-
##### Why don't I see any data in S3 or Glue after enabling a source?
119
-
114
+
{% endfaqitem %}
115
+
{% faqitem Can I configure multiple sources to use the same EMR cluster? %}
116
+
Yes, you can configure multiple sources to use the same EMR cluster. We recommend that the EMR cluster only be used for Data Lakes to ensure there aren't interruptions from non-Data Lakes job.
117
+
{% endfaqitem %}
118
+
{% endfaq %}
119
+
120
+
### Post-Set Up
121
+
{% faq %}
122
+
{% faqitem Why don't I see any data in S3 or Glue after enabling a source? %}
120
123
If you don't see data after enabling a source, check the following:
124
+
- Does the IAM role have the Segment account ID and source IDs as the external IDs?
121
125
- Is the EMR cluster running?
122
126
- Is the correct IAM role and S3 bucket configured in the settings?
123
-
- Does the IAM role have the Segment account ID and source IDs as the external IDs?
124
127
125
128
If all of these look correct and you're still not seeing any data, please [contact the Support team](https://segment.com/help/contact/).
126
-
127
-
##### What are "Segment Output" tables in S3?
128
-
129
+
{% endfaqitem %}
130
+
{% faqitem What are "Segment Output" tables in S3? %}
129
131
The `output` tables are temporary tables Segment creates when loading data. They are deleted after each sync.
130
-
131
-
##### Can I make additional directories in the S3 bucket Data Lakes is using?
132
-
132
+
{% endfaqitem %}
133
+
{% faqitem Can I make additional directories in the S3 bucket Data Lakes is using? %}
133
134
Yes, you can create new directories in S3 without interfering with Segment data.
134
135
Do not modify, or create additional directories with the following names:
135
136
-`logs/`
136
137
-`segment-stage/`
137
138
-`segment-data/`
138
139
-`segment-logs/`
139
-
140
-
##### What does "partitioned" mean in the table name?
141
-
140
+
{% endfaqitem %}
141
+
{% faqitem What does "partitioned" mean in the table name? %}
142
142
`Partitioned` just means that the table has partition columns (day and hour). All tables are partitioned, so you should see this on all table names.
143
-
144
-
##### Why are the Filters, Event Tester and Event Delivery tabs in-app empty?
145
-
146
-
Data Lakes does not currently support these features. Sync history information will be available soon.
147
-
148
-
##### How can I use AWS Spectrum to access Data Lakes tables in Glue, and join it with Redshift data?
149
-
143
+
{% endfaqitem %}
144
+
{% faqitem How can I use AWS Spectrum to access Data Lakes tables in Glue, and join it with Redshift data? %}
150
145
You can use the following command to create external tables in Spectrum to access tables in Glue and join the data with Redshift:
151
146
152
147
Run the `CREATE EXTERNAL SCHEMA` command:
@@ -162,3 +157,5 @@ create external database if not exists;
162
157
Replace:
163
158
-[glue_db_name] = The Glue database created by Data Lakes which is named after the source slug
164
159
-[spectrum_schema_name] = The schema name in Redshift you want to map to
0 commit comments