Making requested changes, moving update-emr-cluster page into the manual setup page

Author: forstisabella

src/connections/storage/data-lakes/data-lakes-manual-setup.md

@@ -5,90 +5,83 @@ title: Configure the Data Lakes AWS Environment
 {% include content/plan-grid.md name="data-lakes" %}
 
 
-The instructions below will guide you through the process required to configure the environment required to begin loading data into your Segment Data Lake. For a more automated process, see [Set Up Segment Data Lakes](/src/connections/storage/catalog/data-lakes/index.md).
+The instructions below will guide you through the process required to configure the environment required to begin loading data into your Segment Data Lake. For a more automated process, see [Set Up Segment Data Lakes](/docs/connections/storage/catalog/data-lakes/index).
 
 As a best practice, Segment recommends that you consult with your network and security teams before you configure your EMR cluster.
 
 ## Step 1 - Create a VPC and an S3 bucket 
 
 In this step, you'll create a Virtual Private Cloud (VPC) to securely launch your AWS resources into and an S3 bucket that will store both the intermediate and final data. 
 
-To create a VPC, follow the instructions outlined in Amazon's documentation, [Create and configure your VPC](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/gsg_create_vpc.html).
+To create a VPC, follow the instructions outlined in Amazon's documentation, [Create and configure your VPC](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/gsg_create_vpc.html){:target="_blank"}.
 
-To create an S3 bucket, see Amazon's [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) instructions.
+To create an S3 bucket, see Amazon's [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html){:target="_blank"} instructions.
 
 > info ""
 > Take note of the S3 bucket name you set in this step, as the rest of the set up flow requires it. 
-<!--- In these instructions, the name is `segment-data-lake`. --->
 
-After creating an S3 bucket, configure a lifecycle rule for the bucket and set it to expire staging data after **14 days**. For instructions on configuring lifecycle rules, see Amazon's documentation, [Setting lifecycle configuration on a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html).
+After you create an S3 bucket, configure a lifecycle rule for the bucket and set it to expire staging data after **14 days**. For instructions on configuring lifecycle rules, see Amazon's documentation, [Setting lifecycle configuration on a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html){:target="_blank"}.
 
-The following lifecycle settings should be applied to your staging data:
+Apply the following lifecycle settings to your staging data:
 * **Expire after:** 14 days
 * **Permanently delete after:** 14 days
 * **Clean up incomplete multipart uploads:** after 14 days
 
-<!--- ![Create a Lifecycle rule to expire staging data after 14 days](images/01_14-day-lifecycle.png) --->
-
 ## Step 2 - Configure an EMR cluster
 
 Segment requires access to an EMR cluster to perform necessary data processing. For best results, start with a small cluster and add more compute resources as required.
 
 ### Configure the hardware and networking configuration
 
-1. Select EMR from the AWS console by navigating to Services > Analytics > EMR.
-2. Click **Create Cluster**, and select **Go to advanced options**.
+1. In the AWS console, navigate to **Services > Analytics > EMR**.
+2. Click **Create Cluster**. On the Create Cluster - Quick Options page, click **Go to advanced options**.
 3. In Advanced Options, on Step 1: Software and Steps, select both the `emr-5.33.0` release and the following applications:
     - Hadoop 2.10.1
     - Hive 2.3.7
     - Hue 4.9.0
     - Spark 2.4.7
     - Pig 0.17.0
- 4. Under the AWS Glue Data Catalog settings, select the following options:
+ 4. In the AWS Glue Data Catalog settings, select the following options:
     - Use for Hive table metadata
     - Use for Spark table metadata
-    <!--- ![Select to use for both Have and Spark table metadata](images/02_hive-spark-table.png) --->
 5. Select **Next** to proceed to Step 2: Hardware.
-6. Under the Networking section, select a Network (the VPC you created in [Step 1](#step-1---create-a-vpc-and-an-s3-bucket)) and EC2 Subnet for your EMR instance. Creating the cluster in a private subnet is more secure, but requires additional configuration, while creating a cluster in a public subnet leaves it accessible from the Internet. Users who create clusters in public subnets can configure strict security groups to prevent unauthorized inbound EMR cluster access. See Amazon's document, [Amazon VPC Options - Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-clusters-in-a-vpc.html) for more information.
+6.In the Networking section, select a Network (the VPC you created in [Step 1](#step-1---create-a-vpc-and-an-s3-bucket)) and EC2 Subnet for your EMR instance.
+
+    Creating the cluster in a private subnet is more secure, but requires additional configuration. Creating the cluster in a public subnet leaves it accessible from the Internet, but requires less up front configuration. 
+  
+    If you create clusters in public subnets, you can configure strict security groups to prevent unauthorized inbound EMR cluster access. See Amazon's document, [Amazon VPC Options - Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-clusters-in-a-vpc.html){:target="_blank"} for more information.
 
 7. In the Cluster Nodes and Instances section, create a cluster that includes the following on-demand nodes:
    - **1** master node
    - **2** core nodes
    - **2** task nodes
-   <!--- ![Configure the number of nodes](images/03_hardware-node-instances.png --->
-    Each of the master, core, and task nodes should meet or exceed the following specifications:
+  
+    Each node should meet or exceed the following specifications:
     * Instance type: mx5.xlarge
     * Number of vCores: 4
     * Memory: 16 GiB
     * EBS Storage: 64 GiB, EBS only storage
 
     For more information about configuring cluster hardware and networking, see Amazon's documentation, [Configure Cluster Hardware and Networking](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances.html).
 
-8. Select **Next** to proceed to Step 3: General Cluster Settings.
+8. Click **Next** to proceed to Step 3: General Cluster Settings.
 
 ### Configure logging
 
 9. On Step 3: General Cluster Settings, configure logging to use the same S3 bucket you configured as the destination for the final data. Once configured, logs are assigned a new prefix and separated from the final processed data.
-<!--- (`segment-data-lakes` in this case) --->
-
-10. Add a new key-value pair to the Tags section, a **vendor** key with a value of **segment**. The IAM policy uses this to provide Segment access to submit jobs in the EMR cluster.
 
-11. Select **Next** to proceed to Step 4: Security.
+10. Add a new key-value pair to the Tags section, a `vendor` key with a value of `segment`. The IAM policy uses this tag to provide Segment access to submit jobs in the EMR cluster.
 
-<!---![Configure logging](images/05_logging.png) --->
+11. Click **Next** to proceed to Step 4: Security.
 
 ### Secure the cluster
 12. On Step 4: Security, in the Security Options section, create or select an **EC2 key pair**.
 13. Choose the appropriate roles in the **EC2 instance profile**.
 14. Expand the EC2 security groups section and select the appropriate security groups for the Master and Core & Task types.
 15. Select **Create cluster**.
 
- <!--- ![Secure the cluster](images/06_secure-cluster.png)
-
-The image uses the default settings. You can make these settings more restrictive, if required. --->
-
 > note ""
-> **NOTE:** If you are updating the EMR cluster for an existing Data Lakes instance, note the EMR cluster ID on the confirmation page.
+> If you update the EMR cluster of existing Data Lakes instance, take note of the EMR cluster ID on the confirmation page.
 
 ## Step 3 - Create an Access Management role and policy
 
@@ -126,7 +119,7 @@ Create a `segment-data-lake-role` role for Segment to assume. Attach the followi
 ```
 
 > note ""
-> **NOTE:** Replace the `ExternalID` list with the Segment `WorkspaceID` that contains the sources to sync to the Data Lake.
+> Replace the `ExternalID` list with the Segment `WorkspaceID` that contains the sources to sync to the Data Lake.
 
 ### IAM Policy
 
@@ -216,7 +209,7 @@ Add a policy to the role created above to give Segment access to the relevant Gl
 ```
 
 > note ""
-> **NOTE:** The policy above grants full access to Athena, but the individual Glue and S3 policies determine which table will be queried. Segment queries for debugging purposes, and will notify you before running any queries.
+> The policy above grants full access to Athena, but the individual Glue and S3 policies determine which table is queried. Segment queries for debugging purposes, and notifies you before running any queries.
 
 ## Debugging
 
@@ -226,3 +219,27 @@ Segment requires access to the data and schema for debugging data quality issues
   - Ensure Athena uses Glue as the data catalog. Older accounts may not have this configuration, and may require some additional steps to complete the upgrade. The Glue console typically displays a warning and provides a link to instructions on how to complete the upgrade.
 ![Debugging](images/dl_setup_glueerror.png)
   - An easier alternative is to create a new account that has Athena backed by Glue as the default.
+
+## Updating EMR Clusters
+You can update your existing Data Lake destination to EMR version 5.33.0 by creating a new v5.33.0 cluster in AWS and associating it with your existing Data Lake. After you update the EMR cluster, your Segment Data Lake continues to use the Glue data catalog you initially configured.
+
+When you update an EMR cluster to 5.33.0, you can participate in [AWS Lake Formation](https://aws.amazon.com/lake-formation/?whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc){:target="_blank"}, use dynamic auto-scaling, and experience faster Parquet jobs.  
+
+> info""
+> Your Segment Data Lake does not need to be disabled during the update process, and any ongoing syncs will complete on the old cluster. Any syncs that fail while you are updating the cluster ID field will be restarted on the new cluster.
+
+## Prerequisites
+* An EMR v5.33.0 cluster
+* An existing Segment Data Lakes destination
+
+## Procedure
+1. Open your Segment app workspace and select the Data Lakes destination.
+2. On the Settings tab, select the EMR Cluster ID field and replace the existing ID with the ID of your v5.33.0 EMR cluster. For help finding the cluster ID in AWS, see Amazon's [View cluster status and details](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-view-clusters.html). You don't need to update the Glue Catalog ID, IAM Role ARN, or S3 Bucket name fields.
+3. Click **Save**.
+4. In the AWS EMR console, view the Events tab for your cluster to verify it is receiving data.
+
+You can delete the old EMR cluster from AWS after the following conditions have been met:
+* You have updated all Data Lakes to use the EMR cluster
+* A sync has successfully completed in the new cluster 
+* Data is synced into the new cluster
+* There are no ongoing jobs in the old cluster