Merge pull request #1115 from segmentio/dl_setup_guide

Small updates to manual set up doc

Author: markzegarelli

src/connections/storage/data-lakes/data-lakes-manual-setup.md

@@ -11,7 +11,7 @@ The instructions below will guide you through the process required to configure
 In this step, you'll create the S3 bucket that will store both the intermediate and final data.
 
 > info ""
-> Take note of the S3 bucket name you set in this step, as the rest of the set up flow requires it. In these instructions, `segment-data-lake` is used.
+> Take note of the S3 bucket name you set in this step, as the rest of the set up flow requires it. In these instructions, the name is `segment-data-lake`.
 
 During the set up process, create a Lifecycle rule and set it to expire staging data after **14 days**. For more information, see Amazon's documentation, [How do I create a lifecycle?](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html).
 
@@ -25,44 +25,39 @@ Segment requires access to an EMR cluster to perform necessary data processing.
 
 1. Locate and select EMR from the AWS console.
 2. Click **Create Cluster**, and open the **Advanced Options**.
-3. In the Advanced Options, on Step 1: Software and Steps, ensure the following options are selected, along with the defaults:
+3. In the Advanced Options, on Step 1: Software and Steps, ensure you select the following options, along with the defaults:
    - `Use for Hive table metadata`
    - `Use for Spark table metadata` ![Select to use for both Have and Spark table metadata](images/02_hive-spark-table.png)
-4. In the Networking setup section, select to create the cluster in either a public or private subnet. Creating the cluster in a private subnet is more secure, but requires some additional configuration. Creating a cluster in a public subnet is accessible from the internet. However, you can configure strict security groups to prevent inbound access to the cluster. See Amazon's document, [Amazon VPC Options - Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-clusters-in-a-vpc.html) for more information. As a best practice, Segment recommends that you consult with your network and security before you configure your EMR cluster.
+4. In the Networking setup section, select to create the cluster in either a public or private subnet. Creating the cluster in a private subnet is more secure, but requires additional configuration. Creating a cluster in a public subnet is accessible from the internet. You can configure strict security groups to prevent inbound access to the cluster. See Amazon's document, [Amazon VPC Options - Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-clusters-in-a-vpc.html) for more information. As a best practice, Segment recommends that you consult with your network and security before you configure your EMR cluster.
 5. In the Hardware Configuration section, create a cluster with the nodes listed below. This configuration uses the default **On demand** purchasing option for the instances.
    - **1** master node
    - **2** core nodes
    - **2** task nodes ![Configure the number of nodes](images/03_hardware-node-instances.png)
- 
-For more information about configuring the cluster hardware and networking, see Amazon's document, [Configure Cluster Hardware and Networking](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances.html).
-
-### Enable EMR managed scaling for the Core and Task nodes
 
-On the **Cluster Scaling** settings, select **Use EMR-managed scaling**, and select the following number of task units:
-- Minimum: **2**
-- Maximum: **8**
-- On-demand limit: **8**
-- Maximum Core Node: **2**
+For more information about configuring the cluster hardware and networking, see Amazon's document, [Configure Cluster Hardware and Networking](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances.html).
 
-![Configure the Cluster scaling options](images/04_cluster-scaling.png)
 
 ### Configure logging
 
-On the General Options step, configure logging to use the same S3 bucket you configured as the destination for the final data (`segment-data-lakes` in this case). Once configured, logs will be written to a new prefix, and separated from the final processed data.
+On the General Options step, configure logging to use the same S3 bucket you configured as the destination for the final data (`segment-data-lakes` in this case). Once configured, logs are to a new prefix, and separated from the final processed data.
+
+Set value of the **vendor** tag to `segment`. The IAM policy uses this to provide Segment access to submit jobs in the EMR cluster.
 
-Set value of the **vendor** tag to `segment`.
 
 ![Configure logging](images/05_logging.png)
 
 ### Secure the cluster
 
-On the Security step, ensure that the following steps have been completed:
+On the Security step, be sure to complete the following steps:
 1. Create or select an **EC2 key pair**.
 2. Choose the appropriate roles in the **EC2 instance profile**.
 3. Select the appropriate security groups for the Master and Core & Task types.
 
 ![Secure the cluster](images/06_secure-cluster.png)
 
+The image uses the default settings. You can make these settings more restrictive, if required.
+
+
 ## Step 3 - Create an Access Management role and policy
 
 The following steps provide examples of the IAM Role and IAM Policy.
@@ -104,7 +99,7 @@ Create a `segment-data-lake-role` role for Segment to assume. Attach the followi
 
 ### IAM Policy
 
-Add a policy to the role created above to give Segment access to the relevant Glue databases and tables, EMR cluster, and S3
+Add a policy to the role created above to give Segment access to the relevant Glue databases and tables, EMR cluster, and S3.
 
 ```json
 {
@@ -162,7 +157,7 @@ Add a policy to the role created above to give Segment access to the relevant Gl
             "Effect": "Allow",
             "Action": "*",
             "Resource": [
-                "arn:aws:s3:::$BUCKET_NAME/*", 
+                "arn:aws:s3:::$BUCKET_NAME/*",
                 "arn:aws:s3:::$BUCKET_NAME"
             ]
         },
@@ -174,19 +169,29 @@ Add a policy to the role created above to give Segment access to the relevant Gl
             "Resource": [
                 "*"
             ]
+        },
+        {
+            "Sid": "",
+            "Effect": "Allow",
+            "Action": "iam:PassRole",
+            "Resource": [
+                "arn:aws:iam::$ACCOUNT_ID:role/EMR_DefaultRole",
+                "arn:aws:iam::$ACCOUNT_ID:role/EMR_AutoScaling_DefaultRole",
+                "arn:aws:iam::$ACCOUNT_ID:role/EMR_EC2_DefaultRole"
+            ]
         }
     ]
 }
 ```
 
 > note ""
-> **NOTE:** The policy above grants full access to Athena, but the individual Glue and S3 policies decide which table can be queried. Segment queries only for debugging purposes, and will notify you be for running any queries.
+> **NOTE:** The policy above grants full access to Athena, but the individual Glue and S3 policies decide which table is queryable. Segment queries for debugging purposes, and will notify you be for running any queries.
 
 ## Debugging
 
 Segment requires access to the data and schema for debugging data quality issues. The modes available for debugging are:
-- Access the individual objects stored in S3 and the associated schema in order to understand data discrepancies
+- Access the individual objects stored in S3 and the associated schema to understand data discrepancies
 - Run an Athena query on the underlying data stored in S3
   - Ensure Athena uses Glue as the data catalog. Older accounts may not have this configuration, and may require some additional steps to complete the upgrade. The Glue console typically displays a warning and provides a link to instructions on how to complete the upgrade.
-  - An easier alternative is to create a new account that has Athena backed by Glue as the default. 
-
+![Debugging](images/dl_setup_glueerror.png)
+  - An easier alternative is to create a new account that has Athena backed by Glue as the default.