|
| 1 | +--- |
| 2 | +title: Open-source components and versions - Azure HDInsight 5.x |
| 3 | +description: Learn about the open-source components and versions in Azure HDInsight 5.x |
| 4 | +ms.service: hdinsight |
| 5 | +ms.topic: conceptual |
| 6 | +ms.date: 03/16/2023 |
| 7 | +--- |
| 8 | + |
| 9 | +# HDInsight 5.x component versions |
| 10 | + |
| 11 | +In this article, you learn about the open-source components and their versions in Azure HDInsight 5.x. |
| 12 | + |
| 13 | +## Public preview |
| 14 | + |
| 15 | +From February 27, 2023 we have started rolling out a new version of HDInsight 5.1, this version is backward compatible with HDInsight 4.0. and 5.0. All new open-source releases added as incremental releases on HDInsight 5.1. |
| 16 | + |
| 17 | +**Only Kafka and HBase clusters are supported now.** |
| 18 | + |
| 19 | +## Open-source components available with HDInsight version 5.x |
| 20 | + |
| 21 | +The Open-source component versions associated with HDInsight 5.1 listed in the following table. |
| 22 | + |
| 23 | +| Component | HDInsight 5.1 | HDInsight 5.0 | |
| 24 | +|------------------|---------------|---------------| |
| 25 | +| Apache Spark | 3.3 * | 3.1.3 | |
| 26 | +| Apache Hive | 3.1.2 * | 3.1.2 | |
| 27 | +| Apache Kafka | 3.2.0 ** | 2.4.1 | |
| 28 | +| Apache Hadoop with YARN | 3.3.4 * | 3.1.1 | |
| 29 | +| Apache Tez | 0.9.1 * | 0.9.1 | |
| 30 | +| Apache Pig | 0.17.0 * | 0.16.1 | |
| 31 | +| Apache Ranger | 2.1.0 * | 1.1.0 | |
| 32 | +| Apache HBase | 2.4.11 ** | - | |
| 33 | +| Apache Sqoop | 1.5.0 * | 1.5.0 | |
| 34 | +| Apache Oozie | 5.2.1 * | 4.3.1 | |
| 35 | +| Apache Zookeeper | 3.6.3 * | 3.4.6 | |
| 36 | +| Apache Livy | 0.7.1 * | 0.5 | |
| 37 | +| Apache Ambari | 2.7.0 ** | 2.7.0 | |
| 38 | +| Apache Zeppelin | 0.10.0 * | 0.8.0 | |
| 39 | +| Apache Phoenix | 5.1.2 ** | - | |
| 40 | + |
| 41 | +\* Under development/Planned |
| 42 | + |
| 43 | +** Public Preview |
| 44 | + |
| 45 | +> [!NOTE] |
| 46 | +> ESP isn't supported for Kafka and HBase in this release. |
| 47 | +
|
| 48 | +### Spark versions supported in Azure HDInsight |
| 49 | + |
| 50 | +Apache Spark versions supported in Azure HDIinsight |
| 51 | + |
| 52 | +|Apache Spark version on HDInsight|Release date|Release stage|End of life announcement date|End of standard support|End of basic support| |
| 53 | +|--|--|--|--|--|--| |
| 54 | +|2.4|July 8, 2019|End of Life Announced (EOLA)| Feb10,2023| Aug 10,2023|Feb 10,2024| |
| 55 | +|3.1|March 11,2022|GA |-|-|-| |
| 56 | +|3.3|To be announced for Public Preview|-|-|-|-| |
| 57 | + |
| 58 | +### Apache Spark 2.4 to Spark 3.x Migration Guides |
| 59 | + |
| 60 | +Spark 2.4 to Spark 3.x Migration Guides see [here](https://spark.apache.org/docs/latest/migration-guide.html). |
| 61 | + |
| 62 | +## HDInsight version 5.0 |
| 63 | + |
| 64 | +Starting from June 1, 2022, we have started rolling out a new version of HDInsight 5.0, this version is backward compatible with HDInsight 4.0. All new open-source releases will be added as incremental releases on HDInsight 5.0. |
| 65 | + |
| 66 | + |
| 67 | +### Spark |
| 68 | + |
| 69 | +:::image type="content" source="./media/hdinsight-release-notes/spark-3-1-for-hdi-5-1.png" alt-text="Screenshot of Spark 3.1 for HDI 5.1"::: |
| 70 | + |
| 71 | +> [!NOTE] |
| 72 | +> * If you are using Azure User Interface to create a Spark Cluster for HDInsight, you will see from the dropdown list an additional version Spark 3.1.(HDI 5.0) along with the older versions. This version is a renamed version of Spark 3.1.(HDI 4.0) and it is backward compatible. |
| 73 | +> * This is only a UI level change, which doesn’t impact anything for the existing users and users who are already using the ARM template to build their clusters. |
| 74 | +> * For backward compatibility, ARM supports creating Spark 3.1 with HDI 4.0 and 5.0 versions which maps to same versions Spark 3.1 (HDI 5.0) |
| 75 | +> * Spark 3.1 (HDI 5.0) cluster comes with HWC 2.0 which works well together with Interactive Query (HDI 5.0) cluster. |
| 76 | +
|
| 77 | +### Interactive Query |
| 78 | + |
| 79 | +:::image type="content" source="./media/hdinsight-release-notes/interactive-query-3-1-for-hdi-5-1.png" alt-text="Screenshot of interactive query 3.1 for HDI 5.1"::: |
| 80 | + |
| 81 | +> [!NOTE] |
| 82 | +> * If you are creating an Interactive Query Cluster, you will see from the dropdown list another version as Interactive Query 3.1 (HDI 5.0). |
| 83 | +> * If you are going to use Spark 3.1 version along with Hive which require ACID support via Hive Warehouse Connector (HWC). You need to select this version Interactive Query 3.1 (HDI 5.0). |
| 84 | +
|
| 85 | +### Kafka |
| 86 | + |
| 87 | +Current ARM template supports HDI 5.0 for Kafka 2.4.1 |
| 88 | + |
| 89 | +`HDI Version '5.0' is supported for clusterType "Kafka" and component Version '2.4'.` |
| 90 | + |
| 91 | +We have fixed the arm templated issue. |
| 92 | + |
| 93 | +### Upcoming version upgrades. |
| 94 | +HDInsight team is working on upgrading other open-source components. |
| 95 | + |
| 96 | +1. Spark 3.2.0 |
| 97 | +1. Kafka 3.2.1 |
| 98 | +1. HBase 2.4.11 |
| 99 | + |
| 100 | + |
| 101 | +## Next steps |
| 102 | + |
| 103 | +- [Cluster setup for Apache Hadoop, Spark, and more on HDInsight](hdinsight-hadoop-provision-linux-clusters.md) |
| 104 | +- [Enterprise Security Package](./enterprise-security-package.md) |
| 105 | +- [Work in Apache Hadoop on HDInsight from a Windows PC](hdinsight-hadoop-windows-tools.md) |
0 commit comments