You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/connections/storage/data-lakes/data-lakes-manual-setup.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ The instructions below will guide you through the process required to configure
11
11
In this step, you'll create the S3 bucket that will store both the intermediate and final data.
12
12
13
13
> info ""
14
-
> Take note of the S3 bucket name you set in this step, as the rest of the set up flow requires it. In these instructions, `segment-data-lake` is used.
14
+
> Take note of the S3 bucket name you set in this step, as the rest of the set up flow requires it. In these instructions, the name is `segment-data-lake`.
15
15
16
16
During the set up process, create a Lifecycle rule and set it to expire staging data after **14 days**. For more information, see Amazon's documentation, [How do I create a lifecycle?](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html).
17
17
@@ -25,10 +25,10 @@ Segment requires access to an EMR cluster to perform necessary data processing.
25
25
26
26
1. Locate and select EMR from the AWS console.
27
27
2. Click **Create Cluster**, and open the **Advanced Options**.
28
-
3. In the Advanced Options, on Step 1: Software and Steps, ensure the following options are selected, along with the defaults:
28
+
3. In the Advanced Options, on Step 1: Software and Steps, ensure you select the following options, along with the defaults:
29
29
-`Use for Hive table metadata`
30
30
-`Use for Spark table metadata`
31
-
4. In the Networking setup section, select to create the cluster in either a public or private subnet. Creating the cluster in a private subnet is more secure, but requires some additional configuration. Creating a cluster in a public subnet is accessible from the internet. However, you can configure strict security groups to prevent inbound access to the cluster. See Amazon's document, [Amazon VPC Options - Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-clusters-in-a-vpc.html) for more information. As a best practice, Segment recommends that you consult with your network and security before you configure your EMR cluster.
31
+
4. In the Networking setup section, select to create the cluster in either a public or private subnet. Creating the cluster in a private subnet is more secure, but requires additional configuration. Creating a cluster in a public subnet is accessible from the internet. You can configure strict security groups to prevent inbound access to the cluster. See Amazon's document, [Amazon VPC Options - Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-clusters-in-a-vpc.html) for more information. As a best practice, Segment recommends that you consult with your network and security before you configure your EMR cluster.
32
32
5. In the Hardware Configuration section, create a cluster with the nodes listed below. This configuration uses the default **On demand** purchasing option for the instances.
33
33
-**1** master node
34
34
-**2** core nodes
@@ -39,23 +39,23 @@ For more information about configuring the cluster hardware and networking, see
39
39
40
40
### Configure logging
41
41
42
-
On the General Options step, configure logging to use the same S3 bucket you configured as the destination for the final data (`segment-data-lakes` in this case). Once configured, logs will be written to a new prefix, and separated from the final processed data.
42
+
On the General Options step, configure logging to use the same S3 bucket you configured as the destination for the final data (`segment-data-lakes` in this case). Once configured, logs are to a new prefix, and separated from the final processed data.
43
43
44
-
Set value of the **vendor** tag to `segment`. This is used in the IAM policy to provide Segment access to submit jobs in the EMR cluster.
44
+
Set value of the **vendor** tag to `segment`. The IAM policy uses this to provide Segment access to submit jobs in the EMR cluster.
45
45
46
46
47
47

48
48
49
49
### Secure the cluster
50
50
51
-
On the Security step, ensure that the following steps have been completed:
51
+
On the Security step, be sure to complete the following steps:
52
52
1. Create or select an **EC2 key pair**.
53
53
2. Choose the appropriate roles in the **EC2 instance profile**.
54
54
3. Select the appropriate security groups for the Master and Core & Task types.
55
55
56
56

57
57
58
-
The image uses the default settings, however these settings can be made more restrictive, if required.
58
+
The image uses the default settings. You can make these settings more restrictive, if required.
59
59
60
60
61
61
## Step 3 - Create an Access Management role and policy
@@ -185,12 +185,12 @@ Add a policy to the role created above to give Segment access to the relevant Gl
185
185
```
186
186
187
187
> note ""
188
-
> **NOTE:** The policy above grants full access to Athena, but the individual Glue and S3 policies decide which table can be queried. Segment queries only for debugging purposes, and will notify you be for running any queries.
188
+
> **NOTE:** The policy above grants full access to Athena, but the individual Glue and S3 policies decide which table is queryable. Segment queries for debugging purposes, and will notify you be for running any queries.
189
189
190
190
## Debugging
191
191
192
192
Segment requires access to the data and schema for debugging data quality issues. The modes available for debugging are:
193
-
- Access the individual objects stored in S3 and the associated schema in order to understand data discrepancies
193
+
- Access the individual objects stored in S3 and the associated schema to understand data discrepancies
194
194
- Run an Athena query on the underlying data stored in S3
195
195
- Ensure Athena uses Glue as the data catalog. Older accounts may not have this configuration, and may require some additional steps to complete the upgrade. The Glue console typically displays a warning and provides a link to instructions on how to complete the upgrade.
0 commit comments