You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Open-source components and versions - Azure HDInsight 5.1
3
+
description: Learn about the open-source components and versions in Azure HDInsight 5.1
4
+
ms.service: hdinsight
5
+
ms.topic: conceptual
6
+
ms.date: 02/28/2023
7
+
---
8
+
9
+
# HDInsight 5.1 component versions
10
+
11
+
In this article, you learn about the open-source components and their versions in Azure HDInsight 5.1.
12
+
13
+
From February 27, 2023 we have started rolling out a new version of HDInsight 5.1, this version is backward compatible with HDInsight 4.0. and 5.0. All new open-source releases added as incremental releases on HDInsight 5.1.
14
+
15
+
## Open-source components available with HDInsight version 5.1
16
+
17
+
The Open-source component versions associated with HDInsight 5.1 listed in the following table.
description: Learn about the open-source components and versions in Azure HDInsight.
4
4
ms.service: hdinsight
5
5
ms.topic: conceptual
6
-
ms.date: 02/01/2022
6
+
ms.date: 02/25/2023
7
7
---
8
8
9
9
# Azure HDInsight versions
10
10
11
-
HDInsight bundles open-source components and HDInsight platform into a package that is deployed on a cluster. For more information, see [how HDInsight versioning works](hdinsight-overview-versioning.md).
11
+
HDInsight bundles open-source components and HDInsight platform into a package that deployed on a cluster. For more information, see [how HDInsight versioning works](hdinsight-overview-versioning.md).
12
12
13
13
## Supported HDInsight versions
14
14
15
15
This table lists the versions of HDInsight that are available in the Azure portal and other deployment methods like PowerShell, CLI and the .NET SDK.
16
16
17
17
| HDInsight version | VM OS | Release date| Support type | Support expiration date | Retirement date | High availability |
18
18
| --- | --- | --- | --- | --- | --- | ---|
19
+
|[HDInsight 5.1](hdinsight-51-component-versioning.md)|Ubuntu 18.0.4 LTS |Feb 27, 2022 |[Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions)| Not announced |Not announced| Yes |
19
20
|[HDInsight 5.0](hdinsight-50-component-versioning.md)|Ubuntu 18.0.4 LTS |July 01, 2022 |[Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions)| Not announced |Not announced| Yes |
20
21
|[HDInsight 4.0](hdinsight-40-component-versioning.md)|Ubuntu 18.0.4 LTS |September 24, 2018 |[Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions)| Not announced | Not announced |Yes |
21
22
22
23
**Support expiration** means that Microsoft no longer provides support for the specific HDInsight version. You may not be able to create clusters from the Azure portal.
23
24
24
-
**Retirement** means that existing clusters of an HDInsight version continue to run as is. New clusters of this version can't be created through any means, which includes the CLI and SDKs. Other control plane features, such as manual scaling and autoscaling, aren't guaranteed to work after retirement date. Support isn't available for retired versions.
25
+
**Retirement** means that existing clusters of an HDInsight version continue to run as is. You can't create new clusters of this version through any means, which includes the CLI and SDKs. Other control plane features, such as manual scaling and autoscaling, not guaranteed to work after retirement date. Support isn't available for retired versions.
25
26
26
27
## Support options for HDInsight versions
27
28
28
-
Support is defined as a time period that an HDInsight version is supported by Microsoft Customer Service and Support. HDInsight offers two types of support:
29
+
Support defined as a time period that an HDInsight version supported by Microsoft Customer Service and Support. HDInsight offers two types of support:
29
30
-**Standard support**
30
31
-**Basic support**
31
32
@@ -46,7 +47,7 @@ Standard support includes
46
47
47
48
### Basic support
48
49
49
-
Basic support provides limited servicing to the HDInsight Resource provider. HDInsight images and open-source software (OSS) components won't be serviced. Only critical security fixes will be patched on HDInsight clusters.
50
+
Basic support provides limited servicing to the HDInsight Resource provider. HDInsight images and open-source software (OSS) components won't be serviced. Only critical security fixes patched on HDInsight clusters.
50
51
51
52
Basic support includes
52
53
- Continued use of existing clusters.
@@ -78,7 +79,7 @@ Microsoft doesn't encourage creating analytics pipelines or solutions on cluster
78
79
For extra release notes on the latest versions of HDInsight, see [HDInsight release notes](hdinsight-release-notes.md).
79
80
80
81
## Versioning considerations
81
-
- Once a cluster is deployed with an image, that cluster isn't automatically upgraded to newer image version. When you create new clusters, most recent image version will be deployed.
82
+
- Once a cluster deployed with an image, that cluster can't automatically upgrade to newer image version. When you create new clusters, most recent image version deployed.
82
83
- Customers should test and validate that applications run properly when using new HDInsight version.
83
84
- HDInsight reserves the right to change the default version without prior notice. If you have a version dependency, specify the HDInsight version when you create your clusters.
84
85
- HDInsight may retire an OSS component version before retiring the HDInsight version.
Copy file name to clipboardExpand all lines: articles/hdinsight/hdinsight-release-notes.md
+40-60Lines changed: 40 additions & 60 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: Latest release notes for Azure HDInsight. Get development tips and
4
4
ms.custom: references_regions
5
5
ms.service: hdinsight
6
6
ms.topic: conceptual
7
-
ms.date: 12/07/2022
7
+
ms.date: 02/28/2023
8
8
---
9
9
10
10
@@ -15,13 +15,13 @@ This article provides information about the **most recent** Azure HDInsight rele
15
15
## Summary
16
16
17
17
Azure HDInsight is one of the most popular services among enterprise customers for open-source analytics on Azure.
18
-
If you would like to subscribe on release notes, watch releases on [this GitHub repository](https://github.com/hdinsight/release-notes/releases).
18
+
[Subscribe to our release notes](./subscribe-to-hdi-release-notes-repo.md) and watch releases on [this GitHub repository](https://github.com/hdinsight/release-notes/releases).
19
19
20
-
## Release date: December 12, 2022
20
+
## Release date: February 28, 2023
21
21
22
-
This release applies to HDInsight 4.0. and 5.0HDInsight release is made available to all regions over several days.
22
+
This release applies to HDInsight 4.0. and 5.0, 5.1. HDInsight release will be available to all regions over several days. This release is applicable for image number 2302250400. [How to check the image number?](./view-hindsight-cluster-image-version.md)
23
23
24
-
HDInsight uses safe deployment practices, which involve gradual region deployment. It may take up to 10 business days for a new release or a new version to be available in all regions.
24
+
HDInsight uses safe deployment practices, which involve gradual region deployment. it may take up to 10 business days for a new release or a new version to be available in all regions.
25
25
26
26
**OS versions**
27
27
@@ -32,72 +32,52 @@ For workload specific versions, see [here.](./hdinsight-40-component-versioning.
32
32
33
33
34
34
35
-
***Log Analytics** - Customers can enable classic monitoring to get the latest OMS version 14.19. To remove old versions, disable and enable classic monitoring.
36
-
***Ambari** user auto UI logout due to inactivity. For more information, see [here](./ambari-web-ui-auto-logout.md)
37
-
***Spark** - A new and optimized version of Spark 3.1.3 is included in this release. We tested Apache Spark 3.1.2(previous version) and Apache Spark 3.1.3(current version) using the TPC-DS benchmark. The test was carried out using E8 V3 SKU, for Apache Spark on 1-TB workload. Apache Spark 3.1.3 (current version) outperformed Apache Spark 3.1.2 (previous version) by over 40% in total query runtime for TPC-DS queries using the same hardware specs. The Microsoft Spark team added optimizations available in Azure Synapse with Azure HDInsight. For more information, please refer to [ Speed up your data workloads with performance updates to Apache Spark 3.1.2 in Azure Synapse](https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/speed-up-your-data-workloads-with-performance-updates-to-apache/ba-p/2769467)
35
+
**HDInsight 5.1**
38
36
39
-
37
+
We have started rolling out a new version of HDInsight 5.1. All new open-source releases added as incremental releases on HDInsight 5.1.
40
38
41
-
* Qatar Central
42
-
* Germany North
43
-
44
-
45
-
46
-
* HDInsight has moved away from Azul Zulu Java JDK 8 to Adoptium Temurin JDK 8, which supports high-quality TCK certified runtimes, and associated technology for use across the Java ecosystem.
47
-
48
-
* HDInsight has migrated to reload4j. The log4j changes are applicable to
49
-
50
-
* Apache Hadoop
51
-
* Apache Zookeeper
52
-
* Apache Oozie
53
-
* Apache Ranger
54
-
* Apache Sqoop
55
-
* Apache Pig
56
-
* Apache Ambari
57
-
* Apache Kafka
58
-
* Apache Spark
59
-
* Apache Zeppelin
60
-
* Apache Livy
61
-
* Apache Rubix
62
-
* Apache Hive
63
-
* Apache Tez
64
-
* Apache HBase
65
-
* OMI
66
-
* Apache Pheonix
39
+
For more information, see [HDInsight 5.1.0 version](./hdinsight-51-component-versioning.md)
67
40
68
41
69
42
70
-
HDInsight will implement TLS1.2 going forward, and earlier versions will be updated on the platform. If you're running any applications on top of HDInsight and they use TLS 1.0 and 1.1, upgrade to TLS 1.2 to avoid any disruption in services.
43
+
**Kafka 3.2.0 Upgrade (Preview)**
71
44
72
-
For more information, see [How to enable Transport Layer Security (TLS)](/mem/configmgr/core/plan-design/security/enable-tls-1-2-client)
45
+
* Kafka 3.2.0 includes several significant new features/improvements.
46
+
* Upgraded Zookeeper to 3.6.3
47
+
* Kafka Streams support
48
+
* Stronger delivery guarantees for the Kafka producer enabled by default.
49
+
* log4j 1.x replaced with reload4j.
50
+
* Send a hint to the partition leader to recover the partition.
51
+
* `JoinGroupRequest` and `LeaveGroupRequest` have a reason attached.
52
+
* Added Broker count metrics8.
53
+
* Mirror Maker2 improvements.
73
54
55
+
**HBase 2.4.11 Upgrade (Preview)**
56
+
* This version has new features such as the addition of new caching mechanism types for block cache, the ability to alter `hbase:meta table` and view the `hbase:meta` table from the HBase WEB UI.
74
57
75
-
58
+
**Phoenix 5.1.2 Upgrade (Preview)**
59
+
* Phoenix version upgraded to 5.1.2 in this release. This upgrade includes the Phoenix Query Server. The Phoenix Query Server proxies the standard Phoenix JDBC driver and provides a backwards-compatible wire protocol to invoke that JDBC driver.
60
+
61
+
**Ambari CVEs**
62
+
* Multiple Ambari CVEs are fixed.
76
63
77
-
End of support for Azure HDInsight clusters on Ubuntu 16.04 LTS from 30 November 2022. HDInsight had begun release of cluster images using Ubuntu 18.04 from June 27, 2021. We recommend our customers who are running clusters using Ubuntu 16.04 is to rebuild their clusters with the latest HDInsight images by 30 November 2022.
78
-
79
-
For more information on how to check Ubuntu version of cluster, see [here](https://learnubuntu.com/check-ubuntu-version)
80
-
81
-
1. Execute the command “lsb_release -a” in the terminal.
82
-
83
-
1. If the value for “Description” property in output is “Ubuntu 16.04 LTS”, then this update is applicable to the cluster.
64
+
84
65
85
-
66
+
End of support for Azure HDInsight clusters on Spark 2.4 February 10, 2024. For more information, see [Spark versions supported in Azure HDInsight](./hdinsight-40-component-versioning.md#spark-versions-supported-in-azure-hdinsight)
86
67
87
-
* Support for Availability Zones selection for Kafka and HBase (write access) clusters.
68
+
## Upcoming Changes
88
69
89
-
## Open source bug fixes
70
+
* Cluster name change limitation
71
+
* The max length of cluster name will be changed to 45 from 59 in Public, Mooncake and Fairfax.
72
+
* Cluster permissions for secure storage
73
+
* Customers can specify (during cluster creation) whether a secure channel should be used for HDInsight cluster nodes to contact the storage account.
74
+
* Non-ESP ABFS clusters [Cluster Permissions for World Readable]
75
+
* Plan to introduce a change in non-ESP ABFS clusters, which restricts non-Hadoop group users from executing Hadoop commands for storage operations. This change to improve cluster security posture. Customers need to plan for the updates.
76
+
* Open-source upgrades
77
+
* Apache Spark 3.3.0 and Hadoop 3.3.4 are under development on HDInsight 5.1 and will include several significant new features, performance and other improvements.
90
78
91
-
**Hive bug fixes**
79
+
> [!NOTE]
80
+
> We advise customers to use to latest versions of HDInsight [Images](./view-hindsight-cluster-image-version.md) as they bring in the best of open source updates, Azure updates and security fixes.
92
81
93
-
|Bug Fixes|Apache JIRA|
94
-
|---|---|
95
-
|[HIVE-26127](https://issues.apache.org/jira/browse/HIVE-26127)| INSERT OVERWRITE error - File Not Found|
96
-
|[HIVE-24957](https://issues.apache.org/jira/browse/HIVE-24957)| Wrong results when subquery has COALESCE in correlation predicate|
97
-
|[HIVE-24999](https://issues.apache.org/jira/browse/HIVE-24999)| HiveSubQueryRemoveRule generates invalid plan for IN subquery with multiple correlations|
98
-
|[HIVE-24322](https://issues.apache.org/jira/browse/HIVE-24322)| If there's direct insert, the attempt ID has to be checked when reading the manifest fails|
99
-
|[HIVE-23363](https://issues.apache.org/jira/browse/HIVE-23363)| Upgrade DataNucleus dependency to 5.2 |
100
-
|[HIVE-26412](https://issues.apache.org/jira/browse/HIVE-26412)| Create interface to fetch available slots and add the default|
101
-
|[HIVE-26173](https://issues.apache.org/jira/browse/HIVE-26173)| Upgrade derby to 10.14.2.0|
102
-
|[HIVE-25920](https://issues.apache.org/jira/browse/HIVE-25920)| Bump Xerce2 to 2.12.2.|
103
-
|[HIVE-26300](https://issues.apache.org/jira/browse/HIVE-26300)| Upgrade Jackson data bind version to 2.12.6.1+ to avoid CVE-2020-36518|
0 commit comments