Skip to content

Commit 42b26df

Browse files
v-dirichardsv-dirichards
authored
Merge pull request #265278 from v-akarnase/patch-25
Update hdinsight-aks-release-notes.md
2 parents 183091a + 9002e11 commit 42b26df

File tree

6 files changed

+113
-45
lines changed

6 files changed

+113
-45
lines changed

‎articles/hdinsight-aks/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ items:
4040
items:
4141
- name: Release notes
4242
href: ./release-notes/hdinsight-aks-release-notes.md
43+
- name: Archived release notes
44+
href: ./release-notes/hdinsight-aks-release-notes-archive.md
4345
- name: Get started
4446
items:
4547
- name: Create a cluster pool and cluster

‎articles/hdinsight-aks/flink/flink-overview.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ Apache Flink clusters in HDInsight on AKS include the following components, they
6161
* [DataStreamAPI](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/overview/#what-is-a-datastream)
6262
* [TableAPI & SQL](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/overview/#table-api--sql).
6363

64-
Refer to the [Roadmap](../whats-new.md#coming-soon) on what's coming soon!
64+
Refer to the [Roadmap](../whats-new.md#roadmap-of-features) on what's coming soon!
6565

6666
## Apache Flink Job Management
6767

‎articles/hdinsight-aks/overview.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ description: An introduction to Azure HDInsight on AKS.
44
ms.custom: references_regions
55
ms.service: hdinsight-aks
66
ms.topic: overview
7-
ms.date: 08/29/2023
7+
ms.date: 02/06/2024
88
---
99

1010
# What is HDInsight on AKS? (Preview)
@@ -56,7 +56,7 @@ The latest version of HDInsight is orchestrated using AKS, which enables the pla
5656

5757
HDInsight on AKS can connect seamlessly with HDInsight. You can reap the benefits of using needed cluster types in a hybrid model. Interoperate with cluster types of HDInsight using the same storage and metastore across both the offerings.
5858

59-
[HDInsight](/azure/hdinsight/) offers Apache Kafka®, Apache HBase® and other analytics workloads in Platform as a Service (PaaS) formfactor.
59+
[HDInsight](/azure/hdinsight/) offers Apache Kafka®, Apache HBase® and other analytics workloads in Platform as a Service (PaaS) form factor.
6060

6161
:::image type="content" source="./media/overview/connectivity-diagram.png" alt-text="Diagram showing connectivity concepts.":::
6262

@@ -84,6 +84,7 @@ For more information, see [HDInsight on AKS security](./concept-security.md).
8484
* West US 2
8585
* West US 3
8686
* East US
87+
* East Asia
8788

8889
> [!Note]
8990
> - The Trino brand and trademarks are owned and managed by the [Trino Software Foundation](https://trino.io/foundation.html). No endorsement by The Trino Software Foundation is implied by the use of these marks.

‎articles/hdinsight-aks/release-notes/hdinsight-aks-release-notes-archive.md

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,52 @@ title: Archived release notes for Azure HDInsight on AKS
33
description: Archived release notes for Azure HDInsight on AKS. Get development tips and details for Trino, Flink, and Spark.
44
ms.service: hdinsight-aks
55
ms.topic: conceptual
6-
ms.date: 08/29/2023
6+
ms.date: 02/07/2024
77
---
88

99
# Azure HDInsight on AKS archived release notes
1010

11-
Now there are no archived release notes available for HDInsight on AKS.
11+
Azure HDInsight on AKS is one of the most popular services among enterprise customers for open-source analytics on Azure. If you would like to subscribe on release notes, watch releases on this [GitHub repository](https://github.com/Azure/HDInsight-on-aks/releases).
12+
13+
## Release date: December 13, 2023
14+
15+
**This hotfix release applies to the following**
16+
17+
- Cluster Pool Version: 1.0
18+
- Cluster Version: 1.0.6
19+
20+
#### Known Issues
21+
22+
- **Secure Tenants User Interface Support**
23+
- This release addresses an issue where the open source component web URLs for HDInsight on AKS Clusters were inaccessible. For applying this fix, please reach out to Azure support to enable this on your subscription/tenant.
24+
25+
**How to apply the Hotfix**
26+
- Recreate Your Cluster
27+
- To apply this hotfix, existing users are required to recreate their [cluster](../quickstart-create-cluster.md) on an existing cluster pool.
28+
29+
### Operating System version
30+
31+
- Mariner OS 2.0
32+
33+
**Workload versions**
34+
35+
|Workload|Version|
36+
| -------- | -------- |
37+
|Trino | 410 |
38+
|Flink | 1.16 |
39+
|Apache Spark | 3.3.1 |
40+
41+
**Supported Java and Scala versions**
42+
43+
|Workload |Java|Scala|
44+
| ----------- | -------- | -------- |
45+
|Trino |Open JDK 17.0.7  |- |
46+
|Flink |Open JDK 11.0.21 |2.12.7 |
47+
|Spark |Open JDK 1.8.0_345  |2.12.15 |
48+
49+
The preview is available in the following [regions](../overview.md#region-availability-public-preview).
50+
51+
If you have any more questions, contact [Azure Support](https://ms.portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview) or refer to the [Support options](../hdinsight-aks-support-help.md) page.
1252

1353

1454

‎articles/hdinsight-aks/release-notes/hdinsight-aks-release-notes.md

Lines changed: 42 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,39 +3,64 @@ title: Release notes for Azure HDInsight on AKS
33
description: Latest release notes for Azure HDInsight on AKS. Get development tips and details for Trino, Flink, Spark, and more.
44
ms.service: hdinsight-aks
55
ms.topic: conceptual
6-
ms.date: 12/20/2023
6+
ms.date: 02/05/2024
77
---
88

99
# Azure HDInsight on AKS release notes
1010

11-
This article provides information about the **most recent** Azure HDInsight on AKS release updates.
11+
This article provides information about the **most recent** Azure HDInsight on AKS release updates. For information on earlier releases, see [Azure HDInsight on AKS archived release notes](./hdinsight-aks-release-notes-archive.md).
1212

1313
## Summary
1414

1515
Azure HDInsight on AKS is a new version of HDInsight, which runs on Kubernetes and brings in the best of the open source on Kubernetes. It's gaining popularity among enterprise customers for open-source analytics on Azure Kubernetes Services.
1616

1717
> [!NOTE]
18-
> To understand about HDInsight on AKS versioning and support, refer to the **[versioning page](../versions.md)**.
18+
> To understand about HDInsight on AKS versioning and support, refer to the **[versioning schema](../versions.md)**.
1919
2020
You can refer to [What's new](../whats-new.md) page for all the details of the features currently in public preview for this release.
2121

2222
## Release Information
2323

24-
### Release date: December 13, 2023
24+
### Release date: February 05, 2024
2525

26-
**This hotfix release applies to the following**
26+
**This release applies to the following**
2727

28-
- Cluster Pool Version: 1.0
29-
- Cluster Version: 1.0.6
28+
- Cluster Pool Version: 1.1
29+
- Cluster Version: 1.1.0
30+
- AKS version: 1.27
3031

31-
#### Known Issues
32+
> [!TIP]
33+
> To create a new HDInsight on AKS cluster on 1.1.0, you are required to create a new cluster pool with version 1.1
34+
35+
### New Features
36+
37+
- [Workload Identity](/azure/aks/workload-identity-overview) is supported by default for cluster pools on 1.1
38+
- Trino clusters support Trino 426 from 1.1.0 release
39+
- HDInsight on AKS now includes all changes up to Trino 426 with several notable improvements provided by the community, learn more about Trino [here](https://trino.io/docs/current/release/release-426.html).
40+
- Trino cluster shape now supports load-based autoscale from 1.1.0 release
41+
- Trino on HDInsight on AKS now supports load-based autoscale making cluster more cost efficient. Learn more about it [here](/azure/hdinsight-aks/hdinsight-on-aks-autoscale-clusters).
42+
- Trino cluster shape adds simplified hive metastore and catalogs configuration
43+
- HDInsight on AKS has simplified external Hive metastore configuration for Trino cluster, you can now specify external metastore in config.properties and enable it for each catalog with single parameter, learn more about enhancements [here](/azure/hdinsight-aks/trino/trino-connect-to-metastore).
44+
- Trino cluster shape adds sharded sql connector
45+
- Flink clusters now support Flink 1.17.0 from HDInsight on AKS 1.1.0 release
46+
- HDInsight on AKS now supports Flink 1.17.0 release, with significant improvements on checkpoints, subtask level flame graph, watermark alignments. Learn more about the Flink 1.17 release [here](https://nightlies.apache.org/flink/flink-docs-release-1.17/release-notes/flink-1.17/)
47+
- Flink [SQL Gateway](https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#sql-client--gateway) is now supported from HDInsight on AKS 1.1.0 release with Flink session clusters
48+
49+
### Bug Fixes & CVEs
50+
51+
- This release includes several critical CVE fixes across the platform and open source components.
52+
- Trino cluster shape excludes system tables from caching automatically
53+
- Trino cluster shape improves Power BI timestamp timezones handling
54+
55+
### Known issues
56+
57+
- **Workload identity limitation:**
58+
- There is a known [limitation](/azure/aks/workload-identity-overview#limitations) when transitioning to workload identity. This is due to the permission-sensitive nature of FIC operations. Users cannot perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion will also fail.
59+
60+
61+
### New regions
62+
- East Asia
3263

33-
- **Secure Tenants User Interface Support**
34-
- This release addresses an issue where the open source component web URLs for HDInsight on AKS Clusters were inaccessible. For applying this fix, please reach out to Azure support to enable this on your subscription/tenant.
35-
36-
**How to apply the Hotfix**
37-
- Recreate Your Cluster
38-
- To apply this hotfix, existing users are required to recreate their [cluster](../quickstart-create-cluster.md) on an existing cluster pool.
3964

4065
### Operating System version
4166

@@ -45,8 +70,8 @@ You can refer to [What's new](../whats-new.md) page for all the details of the f
4570

4671
|Workload|Version|
4772
| -------- | -------- |
48-
|Trino | 410 |
49-
|Flink | 1.16 |
73+
|Trino | 426 |
74+
|Flink | 1.17.0 |
5075
|Apache Spark | 3.3.1 |
5176

5277
**Supported Java and Scala versions**
@@ -59,7 +84,7 @@ You can refer to [What's new](../whats-new.md) page for all the details of the f
5984

6085
The preview is available in the following [regions](../overview.md#region-availability-public-preview).
6186

62-
If you have any more questions, contact [Azure Support](https://ms.portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview) or refer to the [Support options](../hdinsight-aks-support-help.md) page.
87+
If you have any more questions, contact [Azure Support](https://ms.portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview) or refer to the [Support options](../hdinsight-aks-support-help.md) page. If you have product specific feedback, write us on [aka.ms/askhdinsight](https://forms.office.com/pages/responsepage.aspx?id=v4j5cvGGr0GRqy180BHbR6HHTBN7UDpEhLm8BJmDhGJURDhLWEhBVE5QN0FQRUpHWDg4ODlZSDA4RCQlQCN0PWcu).
6388

6489
### Next steps
6590

‎articles/hdinsight-aks/whats-new.md

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: What's new in HDInsight on AKS? (Preview)
33
description: An introduction to new concepts in HDInsight on AKS that aren't in HDInsight.
44
ms.service: hdinsight-aks
55
ms.topic: conceptual
6-
ms.date: 10/10/2023
6+
ms.date: 02/06/2024
77
---
88

99
# What's new in HDInsight on AKS? (Preview)
@@ -38,28 +38,28 @@ The following table list shows the features of HDInsight on AKS that are current
3838
| Logging and Monitoring | Log aggregation in Azure [log analytics](./how-to-azure-monitor-integration.md), for server logs, Cluster and Service metrics via [Managed Prometheus and Grafana](./monitor-with-prometheus-grafana.md), Support server metrics in [Azure monitor](/azure/azure-monitor/overview), Service Status page for monitoring the [Service health](./service-health.md) |
3939
| Auto Scale | Load based [Auto Scale](hdinsight-on-aks-autoscale-clusters.md#create-a-cluster-with-load-based-auto-scale), and Schedule based [Auto Scale](hdinsight-on-aks-autoscale-clusters.md#create-a-cluster-with-schedule-based-auto-scale) |
4040
| Customize and Configure Clusters | Support for [script actions](./manage-script-actions.md) during cluster creation, Support for [library management](./spark/library-management.md), [Service configuration](./service-configuration.md) settings after cluster creation |
41-
| Trino | Support for [Trino catalogs](./trino/trino-add-catalogs.md), [Trino CLI Support](./trino/trino-ui-command-line-interface.md), [DBeaver](./trino/trino-ui-dbeaver.md) support for query submission, Add or remove [plugins](./trino/trino-custom-plugins.md) and [connectors](./trino/trino-connectors.md), Support for [logging query](./trino/trino-query-logging.md) events, Support for [scan query statistics](./trino/trino-scan-stats.md) for any [Connector](./trino/trino-connectors.md) in Trino dashboard, Support for Trino [dashboard](./trino/trino-ui.md) to monitor queries, [Query Caching](./trino/trino-caching.md), Integration with PowerBI, Integration with [Apache Superset](./trino/trino-superset.md), Redash, Support for multiple [connectors](./trino/trino-connectors.md) |
42-
| Flink | Support for Flink native web UI, Flink support with HMS for [DStream](./flink/use-hive-metastore-datastream.md), Submit jobs to the cluster using [REST API and Azure Portal](./flink/flink-job-management.md), Run programs packaged as JAR files via the [Flink CLI](./flink/use-flink-cli-to-submit-jobs.md), Support for persistent Savepoints, Support for update the configuration options when the job is running, Connecting to multiple Azure services: [Azure Cosmos DB](./flink/cosmos-db-for-apache-cassandra.md), [Azure Databricks](./flink/azure-databricks.md), [Azure Data Explorer](./flink/integration-of-azure-data-explorer.md), [Azure Event Hubs](./flink/flink-how-to-setup-event-hub.md), [Azure IoT Hub](./flink/azure-iot-hub.md), [Azure Pipelines](./flink/use-azure-pipelines-to-run-flink-jobs.md), [Azure Data Factory Managed Airflow](./flink/flink-job-orchestration.md), [HDInsight Kafka](./flink/process-and-consume-data.md), Submit jobs to the cluster using [Flink CLI](./flink/use-flink-cli-to-submit-jobs.md) and [CDC](./flink/monitor-changes-postgres-table-flink.md) with Flink |
41+
| Trino | Support for [Trino catalogs](./trino/trino-add-catalogs.md), [Trino CLI Support](./trino/trino-ui-command-line-interface.md), [DBeaver](./trino/trino-ui-dbeaver.md) support for query submission, Add or remove [plugins](./trino/trino-custom-plugins.md) and [connectors](./trino/trino-connectors.md), Support for [logging query](./trino/trino-query-logging.md) events, Support for [scan query statistics](./trino/trino-scan-stats.md) for any [Connector](./trino/trino-connectors.md) in Trino dashboard, Support for Trino [dashboard](./trino/trino-ui.md) to monitor queries, [Query Caching](./trino/trino-caching.md), Integration with Power BI, Integration with [Apache Superset](./trino/trino-superset.md), Redash, Support for multiple [connectors](./trino/trino-connectors.md) |
42+
| Flink | Support for Flink native web UI, Flink support with HMS for [DStream](./flink/use-hive-metastore-datastream.md), Submit jobs to the cluster using [REST API and Azure portal](./flink/flink-job-management.md), Run programs packaged as JAR files via the [Flink CLI](./flink/use-flink-cli-to-submit-jobs.md), Support for persistent Savepoints, Support for update the configuration options when the job is running, Connecting to multiple Azure services: [Azure Cosmos DB](./flink/cosmos-db-for-apache-cassandra.md), [Azure Databricks](./flink/azure-databricks.md), [Azure Data Explorer](./flink/integration-of-azure-data-explorer.md), [Azure Event Hubs](./flink/flink-how-to-setup-event-hub.md), [Azure IoT Hub](./flink/azure-iot-hub.md), [Azure Pipelines](./flink/use-azure-pipelines-to-run-flink-jobs.md), [Azure Data Factory Managed Airflow](./flink/flink-job-orchestration.md), [HDInsight Kafka](./flink/process-and-consume-data.md), Submit jobs to the cluster using [Flink CLI](./flink/use-flink-cli-to-submit-jobs.md) and [CDC](./flink/monitor-changes-postgres-table-flink.md) with Flink |
4343
| Spark | [Jupyter Notebook](./spark/submit-manage-jobs.md), Support for [Delta lake](./spark/azure-hdinsight-spark-on-aks-delta-lake.md) 2.0, Zeppelin Support, Support ATS, Support for Yarn History server interface, Job submission using SSH, Job submission using SDK and [Machine Learning Notebook](./spark/azure-hdinsight-spark-on-aks-delta-lake.md) |
4444

45-
## Coming soon
45+
## Roadmap of Features
4646

47-
| Feature | Estimated release timeline |
48-
| --- | --- |
49-
| Autoscale - Load Based - Trino | Q1 2024 |
50-
| Shuffle aware load based auto scale for Spark | Q2 2024 |
51-
| In Place Upgrade | Q2 2024 |
52-
| Reserved Instance Support | Q2 2024 |
53-
| MSI based authentication for Metastore (SQL) | Q1 2024 |
54-
| Spark 3.4 Support | Q2 2024 |
55-
| Trino 426 Support | Q1 2024 |
56-
| Ranger for RBAC | Q2 2024 |
57-
| App mode support for Flink | Q2 2024 |
58-
| Flink 1.17 Support | Q1 2024 |
59-
| Spark ACID Support | Q1 2024 |
60-
| Configurable SKUs for Headnode, SSH | Q2 2024 |
61-
| Flink SQL Gateway Support | Q2 2024 |
62-
| Private Clusters for HDInsight on AKS | Q2 2024 |
63-
| Ranger Support for Spark SQL | Q2 2024 |
64-
| Ranger ACLs on Storage Layer | Q2 2024 |
65-
| Support for One lake as primary container | Q2 2024 |
47+
| Feature | Estimated release timeline | Status |
48+
| --- | --- | --- |
49+
| Autoscale - Load Based - Trino | Q1 2024 | Completed |
50+
| Shuffle aware load based auto scale for Spark | Q2 2024 | In Progress |
51+
| In Place Upgrade | Q2 2024 | In Progress |
52+
| Reserved Instance Support | Q2 2024 | In Progress |
53+
| MSI based authentication for Metastore (SQL) | Q1 2024 | In Progress |
54+
| Spark 3.4 | Q2 2024 | In Progress |
55+
| Trino 426 | Q1 2024 | Completed |
56+
| Ranger for RBAC | Q2 2024 | In Progress |
57+
| App mode support for Flink | Q2 2024 | In Progress|
58+
| Flink 1.17 | Q1 2024 | Completed |
59+
| Spark ACID Support | Q1 2024 | In Progress|
60+
| Configurable SKUs for Headnode, SSH | Q2 2024 | In Progress|
61+
| Flink SQL Gateway Support | Q1 2024 | Completed |
62+
| Private Clusters for HDInsight on AKS | Q2 2024 |In Progress |
63+
| Ranger Support for Spark SQL | Q2 2024 | In Progress |
64+
| Ranger ACLs on Storage Layer | Q2 2024 | In Progress |
65+
| Support for One lake as primary container | Q2 2024 | In Progress|

0 commit comments

Comments
 (0)