Merge pull request #52914 from MicrosoftDocs/release-ignite-hdinsight

Ignite Merge Down

Author: Cory Fowler

articles/hdinsight/domain-joined/TOC.yml

@@ -4,26 +4,28 @@
   items:
   - name: Overview
     href: apache-domain-joined-introduction.md
-  - name: Plan for domain-joined clusters
+  - name: Plan for ESP clusters
     href: apache-domain-joined-architecture.md
-  - name: Configure domain-joined clusters using Azure AD DS
+  - name: Configure ESP clusters using Azure AD DS
     href: apache-domain-joined-configure-using-azure-adds.md
   - name: Synchronize Azure AD users to an HDInsight cluster
     href: ../hdinsight-sync-aad-users-to-cluster.md
     maintainContext: true
   - name: Configure Hive policies
     href: apache-domain-joined-run-hive.md
+  - name: Configure Kafka policies
+    href: apache-domain-joined-run-kafka.md
     maintainContext: true
-  - name: Use VSCode to link to domain joined cluster
+  - name: Use VSCode to link to ESP cluster
     href: ../hdinsight-for-vscode.md
     maintainContext: true
-  - name: Use IntelliJ to link to domain joined cluster
+  - name: Use IntelliJ to link to ESP cluster
     href: ../spark/apache-spark-intellij-tool-plugin.md
     maintainContext: true
-  - name: Use Eclipse to link to domain joined cluster
+  - name: Use Eclipse to link to ESP cluster
     href: ../spark/apache-spark-eclipse-tool-plugin.md
     maintainContext: true
-  - name: Use Oozie in domain-joined clusters
+  - name: Use Oozie in ESP clusters
     href: hdinsight-use-oozie-domain-joined-clusters.md
 - name: Securing data
   href: ../hdinsight-hadoop-create-linux-clusters-with-secure-transfer-storage.md

articles/hdinsight/domain-joined/apache-domain-joined-architecture.md

@@ -1,16 +1,16 @@
 ---
-title: Domain-joined Azure HDInsight architecture 
-description: Learn how to plan domain-joined HDInsight.
+title: Azure HDInsight architecture with Enterprise Security Package
+description: Learn how to plan HDInsight security with Enterprise Security Package.
 services: hdinsight
 ms.service: hdinsight
 author: omidm1
 ms.author: omidm
 ms.reviewer: jasonh
 ms.custom: hdinsightactive
 ms.topic: conceptual
-ms.date: 05/30/2018
+ms.date: 09/24/2018
 ---
-# Plan Azure domain-joined Hadoop clusters in HDInsight
+# Use Enterprise Security Package in HDInsight
 
 The standard Azure HDInsight cluster is a single-user cluster. It's suitable for most companies that have smaller application teams building large data workloads. Each user can create a dedicated cluster on demand and destroy it when it's not needed anymore. 
 
@@ -23,7 +23,7 @@ The virtual machines (VMs) in HDInsight are domain joined to your provided domai
 
 ## Integrate HDInsight with Active Directory
 
-Open-source Hadoop relies on Kerberos for authentication and security. Therefore, HDInsight cluster nodes are domain-joined to a domain that's managed by Azure AD DS. Kerberos security is configured for the Hadoop components on the cluster. 
+Open-source Hadoop relies on Kerberos for authentication and security. Therefore, HDInsight cluster nodes with Enterprise Security Package (ESP) are joined to a domain that's managed by Azure AD DS. Kerberos security is configured for the Hadoop components on the cluster. 
 
 For each Hadoop component, a service principal is created automatically. A corresponding machine principal is also created for each machine that's joined to the domain. To store these service and machine principals, you must provide an organizational unit (OU) within the domain controller (Azure AD DS), where these principals are placed. 
 
@@ -39,7 +39,7 @@ To summarize, you need to set up an environment with:
 
 The following screenshot shows an OU created in contoso.com. It also shows some of the service principals and machine principals.
 
-![Organization unit for domain-joined HDInsight clusters](./media/apache-domain-joined-architecture/hdinsight-domain-joined-ou.png).
+![Organization unit for HDInsight clusters with ESP](./media/apache-domain-joined-architecture/hdinsight-domain-joined-ou.png).
 
 ## Set up different domain controllers
 HDInsight currently supports only Azure AD DS as the main domain controller that the cluster uses for Kerberos communication. But other complex Active Directory setups are possible, as long as such a setup leads to enabling Azure AD DS for HDInsight access.
@@ -49,17 +49,18 @@ HDInsight currently supports only Azure AD DS as the main domain controller that
 
 Users, groups, and passwords are synchronized from Azure Active Directory (Azure AD). The one-way sync from your Azure AD instance to Azure AD DS enables users to sign in to the cluster by using the same corporate credentials. 
 
-For more information, see [Configure domain-joined HDInsight clusters using Azure AD DS](./apache-domain-joined-configure-using-azure-adds.md).
+For more information, see [Configure HDInsight clusters with ESP using Azure AD DS](./apache-domain-joined-configure-using-azure-adds.md).
 
 ### On-premises Active Directory or Active Directory on IaaS VMs
 
 If you have an on-premises Active Directory instance or more complex Active Directory setups for your domain, you can sync those identities to Azure AD by using Azure AD Connect. You can then enable Azure AD DS on that Active Directory tenant. 
 
 Because Kerberos relies on password hashes, you'll need to [enable password hash sync on Azure AD DS](../../active-directory-domain-services/active-directory-ds-getting-started-password-sync.md). If you're using federation with Active Directory Federation Services (AD FS), you can optionally set up password hash sync as a backup in case your AD FS infrastructure fails. For more information, see [Enable password hash sync with Azure AD Connect sync](../../active-directory/hybrid/how-to-connect-password-hash-synchronization.md). 
 
-Using on-premises Active Directory or Active Directory on IaaS VMs alone, without Azure AD and Azure AD DS, is not a supported configuration for domain-joined HDInsight clusters.
+Using on-premises Active Directory or Active Directory on IaaS VMs alone, without Azure AD and Azure AD DS, is not a supported configuration for HDInsight clusters with ESP.
 
 ## Next steps
-* [Configure domain-joined HDInsight clusters](apache-domain-joined-configure-using-azure-adds.md)
-* [Configure Hive policies for domain-joined HDInsight clusters](apache-domain-joined-run-hive.md)
-* [Manage domain-joined HDInsight clusters](apache-domain-joined-manage.md) 
+
+* [Configure HDInsight clusters with ESP](apache-domain-joined-configure-using-azure-adds.md)
+* [Configure Hive policies for HDInsight clusters with ESP](apache-domain-joined-run-hive.md)
+* [Manage HDInsight  clusters with ESP](apache-domain-joined-manage.md) 

articles/hdinsight/domain-joined/apache-domain-joined-configure-using-azure-adds.md

@@ -1,73 +1,76 @@
 ---
-title: Configure a domain-joined HDInsight cluster by using Azure AD DS
-description: Learn how to set up and configure a domain-joined HDInsight cluster by using Azure Active Directory Domain Services
+title: Configure a HDInsight cluster with Enterprise Security Package by using Azure AD-DS
+description: Learn how to set up and configure a HDInsight Enterprise Security Package cluster by using Azure Active Directory Domain Services
 services: hdinsight
 ms.service: hdinsight
 author: omidm1
 ms.author: omidm
 ms.reviewer: jasonh
 ms.topic: conceptual
-ms.date: 07/17/2018
+ms.date: 09/24/2018
 ---
-# Configure a domain-joined HDInsight cluster by using Azure Active Directory Domain Services
+# Configure a HDInsight cluster with Enterprise Security Package by using Azure Active Directory Domain Services
 
-Domain-joined clusters provide multiuser access on Azure HDInsight clusters. Domain-joined HDInsight clusters are connected to a domain so that domain users can use their domain credentials to authenticate with the clusters and run big data jobs. 
+Enterprise Security Package (ESP) clusters provide multi-user access on Azure HDInsight clusters. HDInsight clusters with ESP are connected to a domain so that domain users can use their domain credentials to authenticate with the clusters and run big data jobs. 
 
-In this article, you learn how to configure a domain-joined HDInsight cluster by using Azure Active Directory Domain Services (Azure AD DS).
+In this article, you learn how to configure a HDInsight cluster with ESP by using Azure Active Directory Domain Services (Azure AD-DS).
 
-## Enable Azure AD DS
+>[!NOTE]
+>ESP is available in HDI 3.6+ for Spark, Interactive, and Hadoop. ESP for HBase cluster types is in preview.
 
-Enabling Azure AD DS is a prerequisite before you can create a domain-joined HDInsight cluster. For more information, see [Enable Azure Active Directory Domain Services using the Azure portal](../../active-directory-domain-services/active-directory-ds-getting-started.md). 
+
+## Enable Azure AD-DS
+
+Enabling Azure AD-DS is a prerequisite before you can create a HDInsight cluster with ESP. For more information, see [Enable Azure Active Directory Domain Services using the Azure portal](../../active-directory-domain-services/active-directory-ds-getting-started.md). 
 
 > [!NOTE]
-> Only tenant administrators have the privileges to create an Azure AD DS instance. If you use Azure Data Lake Storage Gen1 as the default storage for HDInsight, make sure that the default Azure AD tenant for Data Lake Storage Gen1 is same as the domain for the HDInsight cluster. Because Hadoop relies on Kerberos and basic authentication, multi-factor authentication needs to be disabled for users who will access the cluster.
+> Only tenant administrators have the privileges to create an Azure AD-DS instance. If you use Azure Data Lake Storage Gen1 as the default storage for HDInsight, make sure that the default Azure AD tenant for Data Lake Storage Gen1 is same as the domain for the HDInsight cluster. Because Hadoop relies on Kerberos and basic authentication, multi-factor authentication needs to be disabled for users who will access the cluster.
 
-After you provision the Azure AD DS instance, create a service account in Azure Active Directory (Azure AD) with the right permissions. If this service account already exists, reset its password and wait until it syncs to Azure AD DS. This reset will result in the creation of the Kerberos password hash, and it might take up to 30 minutes to sync to Azure AD DS. 
+Secure LDAP is for an Azure AD-DS managed domain. When enabling LDAPS, put the domain name in the subject name or the alternative subject name in the certificate. For more information, see [Configure secure LDAP for an Azure AD-DS managed domain](../../active-directory-domain-services/active-directory-ds-admin-guide-configure-secure-ldap.md).
 
-The service account needs the following privileges:
+## Add managed identity
 
-- Join machines to the domain and place machine principals within the OU that you specify during cluster creation.
-- Create service principals within the OU that you specify during cluster creation.
+After you enabled Azure AD-DS, create a managed identity and assign it to the **HDInsight Domain Services Contributor** role in Azure AD-DS Access control.
 
-> [!NOTE]
-> Because Apache Zeppelin uses the domain name to authenticate the administrative service account, the service account *must* have the same domain name as its UPN suffix for Apache Zeppelin to function properly.
+![Azure Active Directory Domain Services Access control](./media/apache-domain-joined-configure-using-azure-adds/hdinsight-configure-managed-identity.png)
 
-To learn more about OUs and how to manage them, see [Create an OU on an Azure AD DS managed domain](../../active-directory-domain-services/active-directory-ds-admin-guide-create-ou.md). 
+For more information, see [What is managed identities for Azure resources](../../active-directory/managed-identities-azure-resources/overview.md).
 
-Secure LDAP is for an Azure AD DS managed domain. For more information, see [Configure secure LDAP for an Azure AD DS managed domain](../../active-directory-domain-services/active-directory-ds-admin-guide-configure-secure-ldap.md).
+## Create a HDInsight cluster with ESP
 
-## Create a domain-joined HDInsight cluster
+The next step is to create the HDInsight cluster with ESP enabled using Azure AD-DS.
 
-The next step is to create the HDInsight cluster by using Azure AD DS and the service account that you created in the previous section.
+It's easier to place both the Azure AD-DS instance and the HDInsight cluster in the same Azure virtual network. If you choose to put them in different virtual networks, you must peer those virtual networks so that HDInsight VMs have a line of sight to the domain controller for joining the VMs. For more information, see [Virtual network peering](../../virtual-network/virtual-network-peering-overview.md).
 
-It's easier to place both the Azure AD DS instance and the HDInsight cluster in the same Azure virtual network. If you choose to put them in different virtual networks, you must peer those virtual networks so that HDInsight VMs have a line of sight to the domain controller for joining the VMs. For more information, see [Virtual network peering](../../virtual-network/virtual-network-peering-overview.md).
+When you create an HDInsight cluster, you have the option to enable Enterprise Security Package to connect your cluster with Azure AD-DS. 
 
-When you create a domain-joined HDInsight cluster, you must supply the following parameters:
+![Azure HDInsight Security and networking](./media/apache-domain-joined-configure-using-azure-adds/hdinsight-create-cluster-security-networking.png)
 
-- **Domain name**: The domain name that's associated with Azure AD DS. An example is contoso.onmicrosoft.com.
+Once you enable ESP, common misconfigurations related to Azure AD-DS will be automatically detected and validated.
 
-- **Domain user name**: The service account in the Azure ADDS DC managed domain that you created in the previous section. An example is [email protected]. This domain user will be the administrator of this HDInsight cluster.
+![Azure HDInsight Enterprise security package domain validation](./media/apache-domain-joined-configure-using-azure-adds/hdinsight-create-cluster-esp-domain-validate.png)
 
-- **Domain password**: The password of the service account.
+Early detection saves time by allowing you to fix errors before creating the cluster.
 
-- **Organizational unit**: The distinguished name of the OU that you want to use with the HDInsight cluster. An example is OU=HDInsightOU,DC=contoso,DC=onmicrosoft,DC=com. If this OU does not exist, the HDInsight cluster tries to create the OU by using the privileges that the service account has. For example, if the service account is in the Azure AD DS Administrators group, it has the right permissions to create an OU. Otherwise, you need to create the OU first and give the service account full control over that OU. For more information, see [Create an OU on an Azure AD DS managed domain](../../active-directory-domain-services/active-directory-ds-admin-guide-create-ou.md).
+![Azure HDInsight Enterprise security package failed domain validation](./media/apache-domain-joined-configure-using-azure-adds/hdinsight-create-cluster-esp-domain-validate-failed.png)
 
-    > [!IMPORTANT]
-    > Include all of the DCs, separated by commas, after the OU (for example, OU=HDInsightOU,DC=contoso,DC=onmicrosoft,DC=com).
+When you create a HDInsight cluster with ESP, you must supply the following parameters:
+
+- **Cluster admin user**: Choose an admin for your cluster from your synced Azure AD-DS.
+
+- **Cluster access groups**: The security groups whose users you want to sync to the cluster should be synced and available in Azure AD-DS. For example, HiveUsers. If you want to specify multiple user groups, separate them by semicolon ‘;’. The group(s) must exist in the directory prior to provisioning. For more information, see [Create a group and add members in Azure Active Directory](../../active-directory/fundamentals/active-directory-groups-create-azure-portal.md). If the group does not exist, an error occurs: "Group HiveUsers not found in the Active Directory."
 
 - **LDAPS URL**: An example is ldaps://contoso.onmicrosoft.com:636.
 
     > [!IMPORTANT]
     > Enter the complete URL, including "ldaps://" and the port number (:636).
 
-- **Access user group**: The security groups whose users you want to sync to the cluster. For example, HiveUsers. If you want to specify multiple user groups, separate them by semicolon ‘;’. The group(s) must exist in the directory prior to provisioning. For more information, see [Create a group and add members in Azure Active Directory](../../active-directory/fundamentals/active-directory-groups-create-azure-portal.md). If the group does not exist, an error occurs: "Group HiveUsers not found in the Active Directory."
-
 The following screenshot shows the configurations in the Azure portal:
 
-   ![Azure HDInsight domain-joined Active Directory Domain Services configuration](./media/apache-domain-joined-configure-using-azure-adds/hdinsight-domain-joined-configuration-azure-aads-portal.png).
+   ![Azure HDInsight ESP Active Directory Domain Services configuration](./media/apache-domain-joined-configure-using-azure-adds/hdinsight-domain-joined-configuration-azure-aads-portal.png).
 
 
 ## Next steps
-* For configuring Hive policies and running Hive queries, see [Configure Hive policies for domain-joined HDInsight clusters](apache-domain-joined-run-hive.md).
-* For using SSH to connect to domain-joined HDInsight clusters, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](../hdinsight-hadoop-linux-use-ssh-unix.md#domainjoined).
+* For configuring Hive policies and running Hive queries, see [Configure Hive policies for HDInsight clusters with ESP](apache-domain-joined-run-hive.md).
+* For using SSH to connect to HDInsight clusters with ESP, see [Use SSH with Linux-based Hadoop on HDInsight from Linux, Unix, or OS X](../hdinsight-hadoop-linux-use-ssh-unix.md#domainjoined).