Merge pull request #269601 from sreekzz/patch-11

HDI on AKS Feb Release Notes

Author: ttorble

articles/hdinsight-aks/release-notes/hdinsight-aks-release-notes.md

@@ -3,16 +3,21 @@ title: Release notes for Azure HDInsight on AKS
 description: Latest release notes for Azure HDInsight on AKS. Get development tips and details for Trino, Flink, Spark, and more.
 ms.service: hdinsight-aks
 ms.topic: conceptual
-ms.date: 02/05/2024
+ms.date: 03/20/2024
 ---
 
 # Azure HDInsight on AKS release notes
 
-This article provides information about the **most recent** Azure HDInsight on AKS release updates. For information on earlier releases, see [Azure HDInsight on AKS archived release notes](./hdinsight-aks-release-notes-archive.md).
+This article provides information about the **most recent** Azure HDInsight on AKS release updates. For information on earlier releases, see [Azure HDInsight on AKS archived release notes](./hdinsight-aks-release-notes-archive.md). If you would like to subscribe on release notes, watch releases on this [GitHub repository](https://github.com/Azure/HDInsight-on-aks/releases).
 
 ## Summary
 
-Azure HDInsight on AKS is a new version of HDInsight, which runs on Kubernetes and brings in the best of the open source on Kubernetes. It's gaining popularity among enterprise customers for open-source analytics on Azure Kubernetes Services.
+HDInsight on AKS is a modern, reliable, secure, and fully managed Platform as a Service (PaaS) that runs on Azure Kubernetes Service (AKS). HDInsight on AKS allows you to deploy popular Open-Source Analytics workloads like Apache Spark™, Apache Flink®️, and Trino without the overhead of managing and monitoring containers.
+
+You can build end-to-end, petabyte-scale Big Data applications span streaming through Apache Flink, data engineering and machine learning using Apache Spark, and Trino's powerful query engine.
+
+All these capabilities combined with HDInsight on AKS’s strong developer focus enable enterprises and digital natives with deep technical expertise to build and operate applications that are right fit for their needs. HDInsight on AKS allows developers to access all the rich configurations provided by open-source software and the extensibility to seamlessly include other ecosystem offerings. This offering empowers developers to test and tune their applications to extract the best performance at optimal cost.
+
 
 > [!NOTE]
 > To understand about HDInsight on AKS versioning and support, refer to the **[versioning schema](../versions.md)**.
@@ -21,45 +26,40 @@ You can refer to [What's new](../whats-new.md) page for all the details of the f
 
 ## Release Information
 
-### Release date: February 05, 2024
+### Release date: March 20, 2024
 
 **This release applies to the following**
 
 - Cluster Pool Version: 1.1
-- Cluster Version: 1.1.0
+- Cluster Version: 1.1.1
 - AKS version: 1.27
 
-> [!TIP]
-> To create a new HDInsight on AKS cluster on 1.1.0, you are required to create a new cluster pool with version 1.1
 
 ### New Features
 
-- [Workload Identity](/azure/aks/workload-identity-overview) is supported by default for cluster pools on 1.1
-- Trino clusters support Trino 426 from 1.1.0 release
-  - HDInsight on AKS now includes all changes up to Trino 426 with several notable improvements provided by the community, learn more about Trino [here](https://trino.io/docs/current/release/release-426.html).
-- Trino cluster shape now supports load-based autoscale from 1.1.0 release
-  - Trino on HDInsight on AKS now supports load-based autoscale making cluster more cost efficient. Learn more about it [here](/azure/hdinsight-aks/hdinsight-on-aks-autoscale-clusters).
-- Trino cluster shape adds simplified hive metastore and catalogs configuration
-  - HDInsight on AKS has simplified external Hive metastore configuration for Trino cluster, you can now specify external metastore in config.properties and enable it for each catalog with single parameter, learn more about enhancements [here](/azure/hdinsight-aks/trino/trino-connect-to-metastore).
-- Trino cluster shape adds sharded sql connector
-- Flink clusters now support Flink 1.17.0 from HDInsight on AKS 1.1.0 release
-  -   HDInsight on AKS now supports Flink 1.17.0 release, with significant improvements on checkpoints, subtask level flame graph, watermark alignments. Learn more about the Flink 1.17 release [here](https://nightlies.apache.org/flink/flink-docs-release-1.17/release-notes/flink-1.17/)
-- Flink [SQL Gateway](https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/#sql-client--gateway) is now supported from HDInsight on AKS 1.1.0 release with Flink session clusters 
+**Apache Flink Application Mode Cluster**
 
-### Bug Fixes & CVEs
+ Application mode clusters are designed to support dedicated resources for large and long-running jobs. When you have resource-intensive or extensive data processing tasks, you can use the [Application Mode Cluster](https://flink.apache.org/2020/07/14/application-deployment-in-flink-current-state-and-the-new-application-mode/#application-mode). This mode allows you to allocate dedicated resources for specific Apache Flink applications, ensuring that they have the necessary computing power and memory to handle large workloads effectively.  
 
-- This release includes several critical CVE fixes across the platform and open source components.
-- Trino cluster shape excludes system tables from caching automatically
-- Trino cluster shape improves Power BI timestamp timezones handling
+**Private Clusters for HDInsight on AKS**
 
-### Known issues
+With private clusters, and outbound cluster settings you can now control ingress and egress traffic from HDInsight on AKS cluster pools and clusters.
 
-- **Workload identity limitation:**
-  - There is a known [limitation](/azure/aks/workload-identity-overview#limitations) when transitioning to workload identity. This is due to the permission-sensitive nature of FIC operations. Users cannot perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion will also fail.
+- Use Azure Firewall or Network Security Groups (NSGs) to control the egress traffic, when you opt to use outbound cluster pool with load balancer.
+- Use Outbound cluster pool with User defined routing to control egress traffic at the subnet level.
+- Use Private AKS cluster feature - To ensure AKS control plane, or API server has internal IP addresses. The network traffic between AKS Control plane / API server and HDInsight on AKS node pools (clusters) remains on the private network only.
+- Avoid creating public IPs for the cluster. Use private ingress feature on your clusters.
 
+**In place Upgrade**
 
-### New regions
-- East Asia
+Upgrade your clusters and cluster pools with the latest software updates. This means that you can enjoy the latest cluster package hotfixes, security updates, and AKS patches, without recreating clusters.  
+
+
+### Known issues
+
+- **Workload identity limitation:**
+  - There's a known [limitation](/azure/aks/workload-identity-overview#limitations) when transitioning to workload identity. This limitation is due to the permission-sensitive nature of FIC operations. Users can't perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion also fails.
+  - **User Assigned Managed Identities (UAMI)** support – There's a limit of 20 FICs per UAMI. You can only create 20 Federated Credentials on an identity. In HDInsight on AKS cluster, FIC (Federated Identity Credential) and SA have one-to-one mapping and only 20 SAs can be created against an MSI. If you want to create more clusters, then you are required to provide different MSIs to overcome the limitation.
 
 
 ### Operating System version
@@ -88,5 +88,5 @@ If you have any more questions, contact [Azure Support](https://ms.portal.azure.
 
 ### Next steps
 
-- [Azure HDInsight on AKS : Frequently asked questions](../faq.md)
-- [Create a cluster pool and cluster](../quickstart-create-cluster.md)
+- [Azure HDInsight on AKS: Frequently asked questions](../faq.md)
+- [Create cluster pool and cluster](../quickstart-create-cluster.md)

articles/hdinsight-aks/whats-new.md

@@ -3,7 +3,7 @@ title: What's new in HDInsight on AKS? (Preview)
 description: An introduction to new concepts in HDInsight on AKS that aren't in HDInsight.
 ms.service: hdinsight-aks
 ms.topic: conceptual
-ms.date: 02/06/2024
+ms.date: 03/20/2024
 ---
 
 # What's new in HDInsight on AKS? (Preview)
@@ -46,20 +46,20 @@ The following table list shows the features of HDInsight on AKS that are current
 
 |	Feature	|	Estimated release timeline	| Status |
 | --- | --- | --- |
-|	Autoscale - Load Based - Trino	|	Q1 2024	| Completed |
+|	[Autoscale - Load Based - Trino](./hdinsight-on-aks-autoscale-clusters.md)	|	Q1 2024	| Completed |
 |	Shuffle aware load based auto scale for Spark	|	Q2 2024	| In Progress |
 |	In Place Upgrade	|	Q2 2024	| In Progress |
 |	Reserved Instance Support 	|	Q2 2024	| In Progress |
 |	MSI based authentication for Metastore (SQL)	|	Q1 2024	| In Progress |
 |	Spark 3.4	|	Q2 2024	| In Progress |
 |	Trino 426 |	Q1 2024	| Completed |
 |	Ranger for RBAC	|	Q2 2024	| In Progress |
-|	App mode support for Flink	|	Q2 2024	| In Progress|
+|	App mode support for Flink	|	Q1 2024	| Completed |
 |	Flink 1.17 	|	Q1 2024	| Completed |
 |	Spark ACID Support	|	Q1 2024	| In Progress|
 |	Configurable SKUs for Headnode, SSH	|	Q2 2024	| In Progress|
-|	Flink SQL Gateway Support	|	Q1 2024	| Completed |
-|	Private Clusters for HDInsight on AKS	|	Q2 2024	|In Progress |
-|	Ranger Support for Spark SQL	|	Q2 2024	| In Progress |
-|	Ranger ACLs on Storage Layer	|	Q2 2024	| In Progress |
+|	[Flink SQL Gateway Support](./flink/start-sql-client-cli-gateway-mode.md)	|	Q1 2024	| Completed |
+|	Private Clusters for HDInsight on AKS	|	Q1 2024	| Completed |
+|	Ranger Support for Spark SQL	|	Q4 2024	| In Progress |
+|	Ranger ACLs on Storage Layer	|	Q4 2024	| In Progress |
 |	Support for One lake as primary container	|	Q2 2024	| In Progress|