Merge pull request #208954 from sreekzz/patch-99

HDI New Version 5.0

Author: v-ccolin

articles/hdinsight/TOC.yml

@@ -47,6 +47,8 @@ items:
           href: ./hdinsight-overview-versioning.md
         - name: HDInsight versions
           href: ./hdinsight-component-versioning.md
+        - name: HDInsight version 5.0
+          href: ./hdinsight-50-component-versioning.md
         - name: HDInsight version 4.0
           href: ./hdinsight-40-component-versioning.md
         - name: HDInsight version 3.6

articles/hdinsight/hdinsight-50-component-versioning.md

@@ -0,0 +1,78 @@
+---
+title: Open-source components and versions - Azure HDInsight 5.0
+description: Learn about the open-source  components and versions in Azure HDInsight 5.0.
+ms.service: hdinsight
+ms.topic: conceptual
+ms.date: 08/25/2022
+---
+
+# HDInsight 5.0 component versions
+
+In this article, you learn about the open-source components and their versions in Azure HDInsight 5.0.
+
+Starting June 1, 2022, we have started rolling out a new  version of HDInsight 5.0, this version is backward compatible with HDInsight 4.0. All new open-source releases will be added as incremental releases on HDInsight 5.0.
+
+## Open-source components available with HDInsight version 5.0
+
+The Open-source component versions associated with HDInsight 5.0 are listed in the following table.
+
+| Component              | HDInsight 5.0 | HDInsight 4.0 |
+|------------------------|---------------| --------------|
+|Apache Spark | 3.1.2 | 2.4.4, 3.1 |
+|Apache Hive | 3.1.2 | 3.1.2 |
+|Apache Kafka | 2.4.1 | 2.1.1, 2.4.1(Preview) |
+|Apache Hadoop |3.1.1 | 3.1.1 |
+|Apache Tez | 0.9.1 |  0.9.1 |
+|Apache Pig	| 0.16.0 | 0.16.1 |
+|Apache Ranger | 1.1.0 | 1.1.0 |
+|Apache Sqoop | 1.5.0 | 1.5.0 |
+|Apache Oozie | 4.3.1 | 4.3.1 |
+|Apache Zookeeper | 3.4.6 | 3.4.6 |
+|Apache Livy | 0.5 | 0.5 |
+|Apache Ambari | 2.7.0 | 2.7.0 |
+|Apache Zeppelin | 0.8.0 | 0.8.0 |
+
+This table lists certain HDInsight 4.0 cluster types that have retired or will be retired soon.
+
+| Cluster Type                    | Framework version | Support expiration date      | Retirement date |
+|---------------------------------|-------------------|------------------------------|-----------------|
+| HDInsight 4.0 Kafka             | 2.1.0             | Sep 30, 2022                 | Oct 1, 2022     |
+
+## Spark
+
+:::image type="content" source="./media/hdinsight-release-notes/spark-3-1-for-hdi-5-0.png" alt-text="Screenshot of Spark 3.1 for HDI 5.0":::
+
+> [!NOTE]
+> * If you are using Azure User Interface to create a Spark Cluster for HDInsight, you will see from the dropdown list an additional version Spark 3.1.(HDI 5.0) along with the older versions. This version is a renamed version of Spark 3.1.(HDI 4.0) and it is backward compatible.  
+> * This is only an UI level change, which doesn’t impact anything for the existing users and users who are already using the ARM template to build their clusters.
+> * For backward compatibility, ARM supports creating Spark 3.1 with HDI 4.0 and 5.0 versions which maps to same versions Sspark 3.1 (HDI 5.0)
+> * Spark 3.1 (HDI 5.0) cluster comes with HWC 2.0 which works well together with Interactive Query (HDI 5.0) cluster.
+
+## Interactive Query
+
+:::image type="content" source="./media/hdinsight-release-notes/interactive-query-3-1-for-hdi-5-0.png" alt-text="Screenshot of interactive query 3.1 for HDI 5.0":::
+
+> [!NOTE]
+> * If you are creating an Interactive Query Cluster, you will see from the dropdown list an other version as Interactive Query 3.1 (HDI 5.0).
+> * If you are going to use Spark 3.1 version along with Hive which require ACID support via Hive Warehouse Connector (HWC). 
+you need to select this version Interactive Query 3.1 (HDI 5.0).
+
+## Kafka 
+
+**Known Issue**  – Current ARM template supports only 4.0 even though it shows 5.0 image in portal Cluster creation may fail with the following error message if you select version 5.0 in the UI.
+
+`HDI Version'5.0" is not supported for clusterType ''Kafka" and component Version ‘2.4'.,Cluster component version is not applicable for HDI version: 5.0 cluster type: KAFKA (Code: BadRequest)`
+
+We're working on this issue, and a fix will be rolled out shortly.
+
+### Upcoming version upgrades. 
+HDInsight  team is working on upgrading other open-source components.
+1. Spark 3.2.0
+1. Kafka 3.2.1
+1. HBase 2.4.9
+
+## Next steps
+
+- [Cluster setup for Apache Hadoop, Spark, and more on HDInsight](hdinsight-hadoop-provision-linux-clusters.md)
+- [Enterprise Security Package](./enterprise-security-package.md)
+- [Work in Apache Hadoop on HDInsight from a Windows PC](hdinsight-hadoop-windows-tools.md)

articles/hdinsight/hdinsight-component-versioning.md

@@ -1,27 +1,28 @@
 ---
-title: Apache Hadoop components and versions - Azure HDInsight 
-description: Learn about the Apache Hadoop components and versions in Azure HDInsight.
+title: Open-source components and versions - Azure HDInsight 
+description: Learn about the open-source components and versions in Azure HDInsight.
 ms.service: hdinsight
 ms.topic: conceptual
-ms.date: 08/05/2022
+ms.date: 08/25/2022
 ---
 
 # Azure HDInsight versions
 
-HDInsight bundles Apache Hadoop environment components and HDInsight platform into a package that is deployed on a cluster. For more information, see [how HDInsight versioning works](hdinsight-overview-versioning.md).
+HDInsight bundles open-source components and HDInsight platform into a package that is deployed on a cluster. For more information, see [how HDInsight versioning works](hdinsight-overview-versioning.md).
 
 ## Supported HDInsight versions
 
 This table lists the versions of HDInsight that are available in the Azure portal and other deployment methods like PowerShell, CLI and the .NET SDK.
 
 | HDInsight version | VM OS | Release date| Support type | Support expiration date | Retirement date | High availability |
 | --- | --- | --- | --- | --- | --- | ---|
+| [HDInsight 5.0](hdinsight-50-component-versioning.md) |Ubuntu 18.0.4 LTS |July 01, 2022 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | See [HDInsight 5.0](hdinsight-50-component-versioning.md) for date details. | See [HDInsight 5.0](hdinsight-50-component-versioning.md) for date details. |Yes |
 | [HDInsight 4.0](hdinsight-40-component-versioning.md) |Ubuntu 18.0.4 LTS |September 24, 2018 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | See [HDInsight 4.0](hdinsight-40-component-versioning.md) for date details. | See [HDInsight 4.0](hdinsight-40-component-versioning.md) for date details. |Yes |
 | [HDInsight 3.6](hdinsight-36-component-versioning.md) |Ubuntu 16.0.4 LTS |April 4, 2017      | [Basic](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | Standard support expired on June 30, 2021 for all cluster types.<br> Basic support expires on September 30, 2022. See [HDInsight 3.6 component versions](hdinsight-36-component-versioning.md) for cluster type details. |October 1, 2022 |Yes |
 
-**Support expiration** means that Microsoft no longer provides support for the specific HDInsight version. And it may no longer available through the Azure portal for cluster creation.
+**Support expiration** means that Microsoft no longer provides support for the specific HDInsight version. You may not be able to create clusters from the Azure portal.
 
-**Retirement** means that existing clusters of an HDInsight version continue to run as is. New clusters of this version can't be created through any means, which includes the CLI and SDKs. Other control plane features, such as manual scaling and autoscaling, are not guaranteed to work after retirement date. Support isn't available for retired versions.
+**Retirement** means that existing clusters of an HDInsight version continue to run as is. New clusters of this version can't be created through any means, which includes the CLI and SDKs. Other control plane features, such as manual scaling and autoscaling, aren't guaranteed to work after retirement date. Support isn't available for retired versions.
 
 ## Support options for HDInsight versions
 
@@ -33,22 +34,22 @@ Support is defined as a time period that an HDInsight version is supported by Mi
 
 Standard support provides updates and support on HDInsight clusters. Microsoft recommends building solutions using the most recent fully supported version. 
 
-Standard support includes the following:
+Standard support includes
 - Ability to create support requests on HDInsight 4.0 clusters.
 - Support for troubleshooting solutions built on 4.0 clusters. 
 - Requests to restart services or nodes.
 - Root cause analysis investigations on support requests.
 - Root cause analysis or fixes to improve job or query performance.
-- Root cause analysis or fixes to improve customer-initiated changes, e.g., changing service configurations or issues due to custom script actions.
+- Root cause analysis or fixes to improve customer-initiated changes, for example, changing service configurations or issues due to custom script actions.
 - Product updates for critical security fixes until version retirement.
 - Scoped product updates to the HDInsight Resource provider.
 - Selective fixes or changes to HDInsight 4.0 images or open-source software (OSS) component versions.
 
 ### Basic support
 
-Basic support provides limited servicing to the HDInsight Resource provider. HDInsight images and open-source software (OSS) components will not be serviced. Only critical security fixes will be patched on HDInsight clusters. 
+Basic support provides limited servicing to the HDInsight Resource provider. HDInsight images and open-source software (OSS) components won't be serviced. Only critical security fixes will be patched on HDInsight clusters. 
 
-Basic support includes the following:
+Basic support includes
 - Continued use of existing HDInsight 3.6 clusters.
 - Ability for existing HDInsight 3.6 customers to create new 3.6 clusters.
 - Ability to scale HDInsight 3.6 clusters up and down via autoscale or manual scale.
@@ -57,16 +58,16 @@ Basic support includes the following:
 - Ability to create support requests on HDInsight 3.6 clusters.
 - Requests to restart services or nodes.
 
-Basic support does not include the following:
+Basic support doesn't include
 - Fixes or changes to HDInsight 3.6 images or open-source software (OSS) component versions.
 - Support for troubleshooting solutions built on 3.6 clusters. 
 - Adding new features or functionality.
 - Support for advice or ad-hoc queries.
 - Root cause analysis investigations on support requests.
 - Root cause analysis or fixes to improve job or query performance.
-- Root cause analysis or fixes to improve customer-initiated changes, e.g., changing service configurations or issues due to custom script actions.
+- Root cause analysis or fixes to improve customer-initiated changes, for example, changing service configurations or issues due to custom script actions.
 
-Microsoft does not encourage creating analytics pipelines or solutions on clusters in basic support. We recommend migrating existing clusters to the most recent fully supported version. 
+Microsoft doesn't encourage creating analytics pipelines or solutions on clusters in basic support. We recommend migrating existing clusters to the most recent fully supported version. 
 
 ## HDInsight 3.6 to 4.0 Migration Guides
 - [Migrate Apache Spark 2.1 and 2.2 workloads to 2.3 and 2.4](spark/migrate-versions.md).
@@ -76,10 +77,10 @@ Microsoft does not encourage creating analytics pipelines or solutions on cluste
 
 ## Release notes
 
-For additional release notes on the latest versions of HDInsight, see [HDInsight release notes](hdinsight-release-notes.md).
+For extra release notes on the latest versions of HDInsight, see [HDInsight release notes](hdinsight-release-notes.md).
 
 ## Versioning considerations
-- Once a cluster is deployed with an image, that cluster is not automatically upgraded to newer image version. When creating new clusters, most recent image version will be deployed.
+- Once a cluster is deployed with an image, that cluster isn't automatically upgraded to newer image version. When you create new clusters, most recent image version will be deployed.
 - Customers should test and validate that applications run properly when using new HDInsight version.
 - HDInsight reserves the right to change the default version without prior notice. If you have a version dependency, specify the HDInsight version when you create your clusters.
 - HDInsight may retire an OSS component version before retiring the HDInsight version.