Merge pull request #191403 from sreekzz/patch-2

Update hdinsight-release-notes.md

Author: ttorble

articles/hdinsight/hdinsight-release-notes.md

@@ -4,7 +4,7 @@ description: Latest release notes for Azure HDInsight. Get development tips and
 ms.custom: references_regions
 ms.service: hdinsight
 ms.topic: conceptual
-ms.date: 12/27/2021
+ms.date: 03/10/2022
 ---
 # Azure HDInsight release notes
 
@@ -15,15 +15,74 @@ This article provides information about the **most recent** Azure HDInsight rele
 Azure HDInsight is one of the most popular services among enterprise customers for open-source analytics on Azure.
 If you would like to subscribe on release notes, watch releases on [this GitHub repository](https://github.com/hdinsight/release-notes/releases).
 
-## Release date: 12/27/2021
+## Release date: 03/10/2022
 
 This release applies for HDInsight 4.0. HDInsight release is made available to all regions over several days. The release date here indicates the first region release date. If you don't see below changes, wait for the release being live in your region over several days.
 
-The OS versions for this release are:
-- HDInsight 4.0: Ubuntu 18.04.5 LTS
+The OS versions for this release are: 
+- 	HDInsight 4.0: Ubuntu 18.04.5 
 
-HDInsight 4.0 image has been updated to mitigate Log4j vulnerability as described in [Microsoft’s Response to CVE-2021-44228 Apache Log4j 2.](https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/)
+## Spark 3.1 is now generally available
 
-> [!Note]
-> * Any HDI 4.0 clusters created post 27 Dec 2021 00:00 UTC are created with an updated version of the image which mitigates the log4j vulnerabilities. Hence, customers need not patch/reboot these clusters.
-> * For new HDInsight 4.0 clusters created between 16 Dec 2021 at 01:15 UTC and 27 Dec 2021 00:00 UTC, HDInsight 3.6 or in pinned subscriptions after 16 Dec 2021 the patch is auto applied within the hour in which the cluster is created, however customers must then reboot their nodes for the patching to complete (except for Kafka Management nodes, which are automatically rebooted).
+Spark 3.1 is now Generally Available on HDInsight 4.0 release.  This release includes 
+
+* Adaptive Query Execution, 
+* Convert Sort Merge Join to Broadcast Hash Join, 
+* Spark Catalyst Optimizer,
+* Dynamic Partition Pruning, 
+* Customers will be able to create new Spark 3.1 clusters and not Spark 3.0 (preview) clusters.
+
+For more details, see the [Apache Spark 3.1](https://techcommunity.microsoft.com/t5/analytics-on-azure-blog/spark-3-1-is-now-generally-available-on-hdinsight/ba-p/3253679) is now Generally Available on HDInsight - Microsoft Tech Community.
+
+For a complete list of improvements, see the [Apache Spark 3.1 release notes.](https://spark.apache.org/releases/spark-release-3-1-2.html)
+
+For more details on migration, see the [migration guide.](https://spark.apache.org/docs/latest/migration-guide.html)
+
+## Kafka 2.4 is now generally available
+
+Kafka 2.4.1 is now Generally Available.  For more information, please see [Kafka 2.4.1 Release Notes.](http://kafka.apache.org/24/documentation.html) 
+Other features include MirrorMaker 2 availability, new metric category AtMinIsr topic partition, Improved broker start-up time by lazy on demand mmap of index files, More consumer metrics to observe user poll behavior.
+
+## Map Datatype in HWC is now supported in HDInsight 4.0 
+
+This release includes Map Datatype Support for HWC 1.0 (Spark 2.4) Via the spark-shell  application, and all other all spark clients that HWC supports. Following improvements are included like any other data types:
+
+A user can 
+*	Create a Hive table with any column(s) containing Map datatype, insert data into it and read the results from it.
+*	Create an Apache Spark dataframe with Map Type and do batch/stream reads and writes.
+
+### New regions
+
+HDInsight has now expanded its geographical presence to two new regions: China East 3 and China North 3.
+
+### OSS backport changes
+
+OSS backports that are included in Hive including HWC 1.0 (Spark 2.4) which supports Map data type.
+
+### Here are the OSS backported Apache JIRAs for this release:
+
+| Impacted Feature    |   Apache JIRA                                                      |
+|---------------------|--------------------------------------------------------------------|
+| Metastore direct sql queries with IN/(NOT IN) should be split based on max parameters allowed by SQL DB   | [HIVE-25659](https://issues.apache.org/jira/browse/HIVE-25659)     |
+| Upgrade log4j 2.16.0 to 2.17.0                    | [HIVE-25825](https://issues.apache.org/jira/browse/HIVE-25825)     |
+| Update Flatbuffer version                    | [HIVE-22827](https://issues.apache.org/jira/browse/HIVE-22827)     |
+| Support Map data-type natively in Arrow format                    | [HIVE-25553](https://issues.apache.org/jira/browse/HIVE-25553)     |
+| LLAP external client - Handle nested values when the parent struct is null                    | [HIVE-25243](https://issues.apache.org/jira/browse/HIVE-25243)     |
+| Upgrade arrow version to 0.11.0                    | [HIVE-23987](https://issues.apache.org/jira/browse/HIVE-23987)     |
+
+## Deprecation notices
+### Azure Virtual Machine Scale Sets on HDInsight  
+
+HDInsight will no longer use Azure Virtual Machine Scale Sets to provision the clusters, no breaking change is expected. Existing HDInsight clusters on virtual machine scale sets will have no impact, any new clusters on latest images will no longer use Virtual Machine Scale Sets.  
+
+### Scaling of Azure HDInsight HBase workloads will now be supported only using manual scale
+
+Starting from March 01, 2022, HDInsight will only support manual scale for HBase, there's no impact on running clusters.  New HBase clusters won't be able to enable schedule based Autoscaling.  For more information on how to  manually scale your HBase cluster, refer our documentation on [Manually scaling Azure HDInsight clusters](https://docs.microsoft.com/azure/hdinsight/hdinsight-scaling-best-practices)
+
+## HDInsight 3.6 end of support extension 
+
+HDInsight 3.6 end of support is extended until September 30, 2022.
+
+Starting from September 30, 2022, customers can't create new HDInsight 3.6 clusters. Existing clusters will run as is without the support from Microsoft. Consider moving to HDInsight 4.0 to avoid potential system/support interruption.
+
+Customers who are on Azure HDInsight 3.6 clusters will continue to get [Basic support](https://docs.microsoft.com/azure/hdinsight/hdinsight-component-versioning#support-options-for-hdinsight-versions) until September 30, 2022. After September 30, 2022 customers won't be able to create new HDInsight 3.6 clusters.