Merge pull request #292397 from sreekzz/storage-tab-image-change

MSI for SQL DB

Author: PMEds28

articles/hdinsight/TOC.yml

@@ -215,6 +215,8 @@ items:
           href: ./interactive-query/hive-migration-across-storage-accounts.md 
     - name: Manage
       items:
+      - name: Use Managed Identity for SQL Database authentication in Azure HDInsight
+        href: .//use-managed-identity-for-sql-database-authentication-in-azure-hdinsight.md
       - name: Manage clusters using the Apache Ambari web UI
         href: ./hdinsight-hadoop-manage-ambari.md
       - name: Disable auto logout from Ambari Web UI

articles/hdinsight/hadoop/media/apache-hadoop-linux-create-cluster-get-started-portal/azure-portal-cluster-storage.png

@@ -4,13 +4,13 @@ description: Learn how to create HDInsight clusters with your own custom Apache
 ms.service: azure-hdinsight
 ms.custom: hdinsightactive
 ms.topic: how-to
-ms.date: 09/06/2024
+ms.date: 12/27/2024
 ---
 # Set up HDInsight clusters with a custom Ambari DB
 
 Apache Ambari simplifies the management and monitoring of an Apache Hadoop cluster. Ambari provides an easy to use web UI and REST API. Ambari is included on HDInsight clusters, and is used to monitor the cluster and make configuration changes.
 
-In normal cluster creation, as described in other articles such as [Set up clusters in HDInsight](hdinsight-hadoop-provision-linux-clusters.md), Ambari is deployed in an [S0 Azure SQL Database](/azure/azure-sql/database/resource-limits-dtu-single-databases#standard-service-tier) that is managed by HDInsight and is not accessible to users.
+In normal cluster creation, as described in other articles such as [Set up clusters in HDInsight](hdinsight-hadoop-provision-linux-clusters.md), Ambari is deployed in an [S0 Azure SQL Database](/azure/azure-sql/database/resource-limits-dtu-single-databases#standard-service-tier) managed by HDInsight and isn't accessible to users.
 
 The custom Ambari DB feature allows you to deploy a new cluster and setup Ambari in an external database that you manage. The deployment is done with an Azure Resource Manager template. This feature has the following benefits:
 
@@ -25,11 +25,11 @@ The remainder of this article discusses the following points:
 
 ## Custom Ambari DB requirements
 
-You can deploy a custom Ambari DB with all cluster types and versions. Multiple clusters cannot use the same Ambari DB.
+You can deploy a custom Ambari DB with all cluster types and versions. Multiple clusters can't use the same Ambari DB.
 
 The custom Ambari DB has the following other requirements:
 
-- The name of the database cannot contain hyphens or spaces
+- The name of the database can't contain hyphens or spaces
 - You must have an existing Azure SQL DB server and database.
 - The database that you provide for Ambari setup must be empty. There should be no tables in the default dbo schema.
 - The user used to connect to the database should have **SELECT, CREATE TABLE, INSERT, UPDATE, DELETE, ALTER ON SCHEMA and REFERENCES ON SCHEMA** permissions on the database.
@@ -50,7 +50,11 @@ When you host your Apache Ambari DB in an external database, remember the follow
 
 - You're responsible for the extra costs of the Azure SQL DB that holds Ambari.
 - Back up your custom Ambari DB periodically. Azure SQL Database generates backups automatically, but the backup retention time-frame varies. For more information, see [Learn about automatic SQL Database backups](/azure/azure-sql/database/automated-backups-overview).
-- Don't change the custom Ambari DB password after the HDInsight cluster reaches the **Running** state. It is not supported. 
+- Don't change the custom Ambari DB password after the HDInsight cluster reaches the **Running** state. It isn't supported. 
+
+> [!NOTE]
+> You can use Managed Identity to authenticate with SQL database for Ambari. For more information, see [Use Managed Identity for SQL Database authentication in Azure HDInsight](./use-managed-identity-for-sql-database-authentication-in-azure-hdinsight.md)
+
 
 ## Deploy clusters with a custom Ambari DB
 
@@ -69,10 +73,9 @@ az deployment group create --name HDInsightAmbariDBDeployment \
 
 
 > [!WARNING]
-> Please use the following recommended SQL DB and Headnode VM for your HDInsight cluster. Please don't use default Ambari DB (S0) for any production environment. 
+> Use the following recommended SQL DB and Headnode VM for your HDInsight cluster. Don't use default Ambari DB (S0) for any production environment. 
 >
 
-
 ## Database and Headnode sizing
 
 The following table provides guidelines on which Azure SQL DB tier to select based on the size of your HDInsight cluster.

articles/hdinsight/hdinsight-custom-ambari-db.md

@@ -4,7 +4,7 @@ description: Set up Hadoop, Kafka, Spark, or HBase clusters for HDInsight from a
 ms.service: azure-hdinsight
 ms.topic: conceptual
 ms.custom: hdinsightactive, devx-track-azurepowershell, linux-related-content
-ms.date: 04/11/2024
+ms.date: 01/08/2025
 ---
 
 # Set up clusters in HDInsight with Apache Hadoop, Apache Spark, Apache Kafka, and more
@@ -16,9 +16,9 @@ Learn how to set up and configure Apache Hadoop, Apache Spark, Apache Kafka, Int
 A Hadoop cluster consists of several virtual machines (nodes) that are used for distributed processing of tasks. Azure HDInsight handles implementation details of installation and configuration of individual nodes, so you only have to provide general configuration information.
 
 > [!IMPORTANT]
-> HDInsight cluster billing starts once a cluster is created and stops when the cluster is deleted. Billing is pro-rated per minute, so you should always delete your cluster when it is no longer in use. Learn how to [delete a cluster.](hdinsight-delete-cluster.md)
+> HDInsight cluster billing starts once a cluster is created and stops when the cluster is deleted. Billing is pro-rated per minute, so you should always delete your cluster when it's no longer in use. Learn how to [delete a cluster.](hdinsight-delete-cluster.md)
 
-If you're using multiple clusters together, you'll want to create a virtual network, and if you're using a Spark cluster you'll also want to use the Hive Warehouse Connector. For more information, see [Plan a virtual network for Azure HDInsight](./hdinsight-plan-virtual-network-deployment.md) and [Integrate Apache Spark and Apache Hive with the Hive Warehouse Connector](interactive-query/apache-hive-warehouse-connector.md).
+If you're using multiple clusters together, you want to create a virtual network, and if you're using a Spark cluster you also want to use the Hive Warehouse Connector. For more information, see [Plan a virtual network for Azure HDInsight](./hdinsight-plan-virtual-network-deployment.md) and [Integrate Apache Spark and Apache Hive with the Hive Warehouse Connector](interactive-query/apache-hive-warehouse-connector.md).
 
 ## Cluster setup methods
 
@@ -64,7 +64,7 @@ You don't need to specify the cluster location explicitly: The cluster is in the
 Azure HDInsight currently provides the following cluster types, each with a set of components to provide certain functionalities.
 
 > [!IMPORTANT]
-> HDInsight clusters are available in various types, each for a single workload or technology. There is no supported method to create a cluster that combines multiple types, such HBase on one cluster. If your solution requires technologies that are spread across multiple HDInsight cluster types, an [Azure virtual network](../virtual-network/index.yml) can connect the required cluster types.
+> HDInsight clusters are available in various types, each for a single workload or technology. There's no supported method to create a cluster that combines multiple types, such HBase on one cluster. If your solution requires technologies that are spread across multiple HDInsight cluster types, an [Azure virtual network](../virtual-network/index.yml) can connect the required cluster types.
 
 | Cluster type | Functionality |
 | --- | --- |
@@ -82,7 +82,7 @@ Choose the version of HDInsight for this cluster. For more information, see [Sup
 
 With HDInsight clusters, you can configure two user accounts during cluster creation:
 
-* Cluster login username: The default username is *admin*. It uses the basic configuration on the Azure portal. Sometimes it's called "Cluster user," or "HTTP user."
+* Cluster login username: The default username is *admin*. It uses the basic configuration on the Azure portal. Also called as "Cluster user," or "HTTP user."
 * Secure Shell (SSH) username: Used to connect to the cluster through SSH. For more information, see [Use SSH with HDInsight](hdinsight-hadoop-linux-use-ssh-unix.md).
 
 The HTTP username has the following restrictions:
@@ -113,14 +113,14 @@ HDInsight clusters can use the following storage options:
 For more information on storage options with HDInsight, see [Compare storage options for use with Azure HDInsight clusters](hdinsight-hadoop-compare-storage-options.md).
 
 > [!WARNING]
-> Using an additional storage account in a different location from the HDInsight cluster is not supported.
+> Using more storage account in a different location from the HDInsight cluster isn't supported.
 
-During configuration, for the default storage endpoint you specify a blob container of an Azure Storage account or Data Lake Storage. The default storage contains application and system logs. Optionally, you can specify additional linked Azure Storage accounts and Data Lake Storage accounts that the cluster can access. The HDInsight cluster and the dependent storage accounts must be in the same Azure location.
+During configuration, for the default storage endpoint you specify a blob container of an Azure Storage account or Data Lake Storage. The default storage contains application and system logs. Optionally, you can specify more linked Azure Storage accounts and Data Lake Storage accounts that the cluster can access. The HDInsight cluster and the dependent storage accounts must be in the same Azure location.
 
 [!INCLUDE [secure-transfer-enabled-storage-account](includes/hdinsight-secure-transfer.md)]
 
 > [!IMPORTANT]
-> Enabling secure storage transfer after creating a cluster can result in errors using your storage account and is not recommended. It is better to create a new cluster using a storage account with secure transfer already enabled.
+> Enabling secure storage transfer after creating a cluster can result in errors using your storage account and isn't recommended. It's better to create a new cluster using a storage account with secure transfer already enabled.
 
 > [!Note]
 > Azure HDInsight does not automatically transfer, move or copy your data stored in Azure Storage from one region to another.
@@ -132,27 +132,33 @@ You can create optional Hive or Apache Oozie metastores. However, not all cluste
 For more information, see [Use external metadata stores in Azure HDInsight](./hdinsight-use-external-metadata-stores.md).
 
 > [!IMPORTANT]
-> When you create a custom metastore, don't use dashes, hyphens, or spaces in the database name. This can cause the cluster creation process to fail.
+> When you create a custom metastore, don't use dashes, hyphens, or spaces in the database name. This such characters can cause the cluster creation process to fail.
 
 #### SQL database for Hive
 
 If you want to retain your Hive tables after you delete an HDInsight cluster, use a custom metastore. You can then attach the metastore to another HDInsight cluster.
 
 An HDInsight metastore that is created for one HDInsight cluster version can't be shared across different HDInsight cluster versions. For a list of HDInsight versions, see [Supported HDInsight versions](hdinsight-component-versioning.md#supported-hdinsight-versions).
 
+You can use Managed Identity to authenticate with SQL database for Hive. For more information, see [Use Managed Identity for SQL Database authentication in Azure HDInsight](./use-managed-identity-for-sql-database-authentication-in-azure-hdinsight.md)
+
 > [!IMPORTANT]
 > The default metastore provides an Azure SQL Database with a **basic tier 5 DTU limit (not upgradeable)**! Suitable for basic testing purposes. For large or production workloads, we recommend migrating to an external metastore.
 
 #### SQL database for Oozie
 
 To increase performance when using Oozie, use a custom metastore. A metastore can also provide access to Oozie job data after you delete your cluster.
 
+You can use Managed Identity to authenticate with SQL database for Oozie. For more information, see  [Use Managed Identity for SQL Database authentication in Azure HDInsight](./use-managed-identity-for-sql-database-authentication-in-azure-hdinsight.md)
+
 #### SQL database for Ambari
 
-Ambari is used to monitor HDInsight clusters, make configuration changes, and store cluster management information as well as job history. The custom Ambari DB feature allows you to deploy a new cluster and setup Ambari in an external database that you manage. For more information, see [Custom Ambari DB](./hdinsight-custom-ambari-db.md).
+Ambari is used to monitor HDInsight clusters, make configuration changes, and store cluster management information and job history. The custom Ambari DB feature allows you to deploy a new cluster and setup Ambari in an external database that you manage. For more information, see [Custom Ambari DB](./hdinsight-custom-ambari-db.md).
+
+You can use Managed Identity to authenticate with SQL database for Ambari. For more information, see [Use Managed Identity for SQL Database authentication in Azure HDInsight](./use-managed-identity-for-sql-database-authentication-in-azure-hdinsight.md)
 
 > [!IMPORTANT]
-> You cannot reuse a custom Oozie metastore. To use a custom Oozie metastore, you must provide an empty Azure SQL Database when creating the HDInsight cluster.
+> You can't reuse a custom Oozie metastore. To use a custom Oozie metastore, you must provide an empty Azure SQL Database when creating the HDInsight cluster.
 
 ## Security + networking
 
@@ -194,7 +200,7 @@ For more information, see [Managed identities in Azure HDInsight](./hdinsight-ma
 
 :::image type="content" source="./media/hdinsight-hadoop-provision-linux-clusters/azure-portal-cluster-configuration-disk-attach.png" alt-text="HDInsight choose your node size.":::
 
-You're billed for node usage for as long as the cluster exists. Billing starts when a cluster is created and stops when the cluster is deleted. Clusters can't be de-allocated or put on hold.
+You're billed for node usage for as long as the cluster exists. Billing starts when a cluster is created and stops when the cluster is deleted. Clusters can't be deallocated or put on hold.
 
 ### Node configuration
 
@@ -208,7 +214,7 @@ Each cluster type has its own number of nodes, terminology for nodes, and defaul
 
 For more information, see [Default node configuration and virtual machine sizes for clusters](hdinsight-supported-node-configuration.md) in "What are the Hadoop components and versions in HDInsight?"
 
-The cost of HDInsight clusters is determined by the number of nodes and the virtual machines sizes for the nodes.
+The cost of HDInsight clusters determined by the number of nodes and the virtual machines sizes for the nodes.
 
 Different cluster types have different node types, numbers of nodes, and node sizes:
 
@@ -245,9 +251,9 @@ For more information, see [Sizes for virtual machines](/azure/virtual-machines/s
 > The added disks are only configured for node manager local directories and **not for datanode directories**
 
 
-HDInsight cluster comes with pre-defined disk space based on SKU. If you run some large applications, can lead to insufficient disk space, with disk full error -  `LinkId=221672#ERROR_NOT_ENOUGH_DISK_SPACE` and job failures. 
+HDInsight cluster comes with predefined disk space based on SKU. If you run some large applications, can lead to insufficient disk space, with disk full error -  `LinkId=221672#ERROR_NOT_ENOUGH_DISK_SPACE` and job failures. 
 
-More discs can be added to the cluster using the new feature **NodeManager**’s local directory. At the time of Hive and Spark cluster creation, the number of discs can be selected and added to the worker nodes. The selected disk, which will be of size 1TB each, would be part of **NodeManager**'s local directories.
+More discs can be added to the cluster using the new feature **NodeManager**’s local directory. At the time of Hive and Spark cluster creation, the number of discs can be selected and added to the worker nodes. The selected disk, which can be of size 1 TB each, would be part of **NodeManager**'s local directories.
 
 1. From **Configuration + pricing** tab
 1. Select **Enable managed disk** option
@@ -260,18 +266,18 @@ You can verify the number of disks from **Review + create** tab, under **Cluster
 
 HDInsight application is an application, that users can install on a Linux-based HDInsight cluster. You can use applications provided by Microsoft, third parties, or developed by you. For more information, see [Install third-party Apache Hadoop applications on Azure HDInsight](hdinsight-apps-install-applications.md).
 
-Most of the HDInsight applications are installed on an empty edge node.  An empty edge node is a Linux virtual machine with the same client tools installed and configured as in the head node. You can use the edge node for accessing the cluster, testing your client applications, and hosting your client applications. For more information, see [Use empty edge nodes in HDInsight](hdinsight-apps-use-edge-node.md).
+Most of the HDInsight applications are installed on an empty edge node. An empty edge node is a Linux virtual machine with the same client tools installed and configured as in the head node. You can use the edge node for accessing the cluster, testing your client applications, and hosting your client applications. For more information, see [Use empty edge nodes in HDInsight](hdinsight-apps-use-edge-node.md).
 
 ### Script actions
 
-You can install additional components or customize cluster configuration by using scripts during creation. Such scripts are invoked via **Script Action**, which is a configuration option that can be used from the Azure portal, HDInsight Windows PowerShell cmdlets, or the HDInsight .NET SDK. For more information, see [Customize HDInsight cluster using Script Action](hdinsight-hadoop-customize-cluster-linux.md).
+You can install more components or customize cluster configuration by using scripts during creation. Such scripts are invoked via **Script Action**, which is a configuration option that can be used from the Azure portal, HDInsight Windows PowerShell cmdlets, or the HDInsight .NET SDK. For more information, see [Customize HDInsight cluster using Script Action](hdinsight-hadoop-customize-cluster-linux.md).
 
 Some native Java components, like Apache Mahout and Cascading, can be run on the cluster as Java Archive (JAR) files. These JAR files can be distributed to Azure Storage and submitted to HDInsight clusters with Hadoop job submission mechanisms. For more information, see [Submit Apache Hadoop jobs programmatically](hadoop/submit-apache-hadoop-jobs-programmatically.md).
 
 > [!NOTE]
 > If you have issues deploying JAR files to HDInsight clusters, or calling JAR files on HDInsight clusters, contact [Microsoft Support](https://azure.microsoft.com/support/options/).
 > 
-> Cascading is not supported by HDInsight and is not eligible for Microsoft Support. For lists of supported components, see [What's new in the cluster versions provided by HDInsight](hdinsight-component-versioning.md).
+> Cascading not supported by HDInsight and not eligible for Microsoft Support. For lists of supported components, see [What's new in the cluster versions provided by HDInsight](hdinsight-component-versioning.md).
 
 Sometimes, you want to configure the following configuration files during the creation process: