Merge pull request #270218 from nadiacps/nadiacps-patch-1

Nadiacps patch 1

Author: v-dirichards

articles/hdinsight/TOC.yml

@@ -285,6 +285,10 @@ items:
           href: ./hdinsight-known-issues.md
         - name: Conda version regression in a recent HDInsight release
           href: ./hdinsight-known-issues-conda-version-regression.md
+        - name: Cluster Reliability Issues
+          href: ./cluster-reliability-issues.md
+        - name: Kafka 2.4.1 validation error in ARM templates
+          href: ./kafka241-validation-error-arm-templates.md
       - name: Request increase for CPU core quota
         href: ./quota-increase-request.md
       - name: Troubleshoot Apache Ambari Metrics Collector

articles/hdinsight/cluster-reliability-issues.md

@@ -0,0 +1,45 @@
+---
+title: Cluster reliability issue with older images in HDInsight clusters
+description: Cluster reliability issue with older images in HDInsight clusters
+ms.service: hdinsight
+ms.topic: troubleshooting-known-issue
+ms.date: 03/28/2024
+---
+
+# Cluster reliability issue with older images in HDInsight clusters
+
+**Issue published date**: October 13, 2023
+
+As part of the proactive reliability management of Azure HDInsight, we recently found a potential reliability issue on HDInsight clusters that use images dated February 2022 or older.
+
+## Issue background
+
+In HDInsight images dated before March 2022, a known bug was discovered on one particular Azure Linux build. The Microsoft Azure Linux Agent (`waagent`), a lightweight process that manages virtual machines, was unstable and resulted in VM outages. HDInsight clusters that consumed the Azure Linux build have experienced service outages, job failures, and adverse effects on features like IPsec and autoscale.
+
+## Required action
+
+If your cluster was created before March 2022, we advise rebuilding your cluster with the latest HDInsight image. Support for cluster images dated before March 2022 ended on November 10, 2023. These images won't receive security updates, bug fixes, or patches, leaving them highly susceptible to vulnerabilities.
+
+> [!IMPORTANT]  
+> We recommend that you keep your clusters updated to the latest HDInsight version on a regular basis. Using clusters that are based on the latest HDInsight image ensures that they have the latest operating system patches, security patches, bug fixes, and library versions. This practice helps you minimize risk and potential security vulnerabilities.
+
+### FAQ
+
+#### What happens if there's a VM outage in HDInsight clusters that use these affected HDInsight images?
+
+You can't recover such virtual machines through straightforward restarts. The outage could last for several hours and require manual intervention from the Microsoft support team.
+
+#### Is this issue rectified in the latest HDInsight images?
+
+Yes. We fixed this issue on HDInsight images dated on or after March 1, 2022. We advise that you move to the latest stable version to maintain the service-level agreement (SLA) and service reliability.
+
+#### How do I determine the date of the HDInsight image that my clusters are built on?
+
+The last 10 digits in your HDInsight image version indicate the date and time of the image. For example, an image version of 5.0.3000.1.2208310943 indicates a date of August 31, 2022. [Learn how to verify your HDInsight image version](/azure/hdinsight/view-hindsight-cluster-image-version).
+
+
+#### Resources
+
+- [Create HDInsight clusters by using automation](/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-setup-methods)
+- [Supported HDInsight versions](/azure/hdinsight/hdinsight-component-versioning#supported-hdinsight-versions)
+- [Verify your HDInsight image version](/azure/hdinsight/view-hindsight-cluster-image-version)

articles/hdinsight/hdinsight-component-versioning.md

@@ -18,7 +18,7 @@ This table lists the versions of HDInsight that are available in the Azure porta
 | --- | --- | --- | --- | --- | --- | ---|
 | [HDInsight 5.1](./hdinsight-5x-component-versioning.md) |Ubuntu 18.0.4 LTS |November 1, 2023 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | Not announced |Not announced| Yes |
 | [HDInsight 5.0](./hdinsight-5x-component-versioning.md) |Ubuntu 18.0.4 LTS |March 11, 2022 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | Not announced |Not announced| Yes |
-| [HDInsight 4.0](hdinsight-40-component-versioning.md) |Ubuntu 18.0.4 LTS |September 24, 2018 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | Not announced | Not announced |Yes |
+| [HDInsight 4.0](hdinsight-40-component-versioning.md) |Ubuntu 18.0.4 LTS |September 24, 2018 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | March 19, 2025 | March 31, 2025 |Yes |
 
 **Support expiration** means that Microsoft no longer provides support for the specific HDInsight version. You might not be able to create clusters from the Azure portal.
 

articles/hdinsight/hdinsight-known-issues.md

@@ -3,7 +3,7 @@ title: Azure HDInsight known issues
 description: Track known issues for Azure HDInsight, along with troubleshooting steps, actions, and frequently asked questions.
 ms.service: hdinsight
 ms.topic: troubleshooting-known-issue
-ms.date: 10/13/2023
+ms.date: 03/28/2024
 ---
 
 # Azure HDInsight known issues
@@ -14,75 +14,22 @@ For service-level outages or degradation notifications, check the [Azure Service
 
 ## Summary of known issues
 
-Azure HDInsight has the following known issues:
+Azure HDInsight Open known issues:
 
 | HDInsight component | Issue description |
 |---------------------|-------------------|
-| Kafka | [Kafka 2.4.1 validation error in ARM templates](#kafka-241-validation-error-in-arm-templates) |
-| Platform | [Cluster reliability issue with older images in HDInsight clusters](#cluster-reliability-issue-with-older-images-in-hdinsight-clusters)|
+| Kafka | [Kafka 2.4.1 validation error in ARM templates](./kafka241-validation-error-arm-templates.md) |
+| Platform | [Cluster reliability issue with older images in HDInsight clusters](./cluster-reliability-issues.md)|
 
-### Kafka 2.4.1 validation error in ARM templates
 
-**Issue published date**: October 13, 2023
-
-When you're submitting cluster creation requests by using Azure Resource Manager templates (ARM templates), runbooks, PowerShell, the Azure CLI, and other automation tools, you might receive a "BadRequest" error message if you specify `clusterType = "Kafka"`, `HDI version = "5.0"`, and `Kafka version = "2.4.1"`.
-
-#### Troubleshooting steps
-
-When you're using [templates or automation tools](/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-setup-methods) to create HDInsight Kafka clusters, choose `componentVersion = "2.4"`. This value enables you to successfully create a Kafka 2.4.1 cluster in HDInsight 5.0.
-
-#### Resources
-
-- [Create HDInsight clusters by using automation](/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-setup-methods)
-- [Supported HDInsight versions](/azure/hdinsight/hdinsight-component-versioning#supported-hdinsight-versions)
-- [HDInsight Kafka cluster](/azure/hdinsight/kafka/apache-kafka-introduction)
-
-
-### Cluster reliability issue with older images in HDInsight clusters
-
-**Issue published date**: October 13, 2023
-
-As part of the proactive reliability management of Azure HDInsight, we recently found a potential reliability issue on HDInsight clusters that use images dated February 2022 or older.
-
-#### Issue background
-
-In HDInsight images dated before March 2022, a known bug was discovered on one particular Azure Linux build. The Microsoft Azure Linux Agent (`waagent`), a lightweight process that manages virtual machines, was unstable and resulted in VM outages. HDInsight clusters that consumed the Azure Linux build have experienced service outages, job failures, and adverse effects on features like IPsec and autoscale.
-
-#### Required action
-
-If your cluster was created before March 2022, we advise rebuilding your cluster with the latest HDInsight image. Support for cluster images dated before March 2022 ended on November 10, 2023. These images won't receive security updates, bug fixes, or patches, leaving them highly susceptible to vulnerabilities.
-
-> [!IMPORTANT]  
-> We recommend that you keep your clusters updated to the latest HDInsight version on a regular basis. Using clusters that are based on the latest HDInsight image ensures that they have the latest operating system patches, security patches, bug fixes, and library versions. This practice helps you minimize risk and potential security vulnerabilities.
-
-#### FAQ
-
-##### What happens if there's a VM outage in HDInsight clusters that use these affected HDInsight images?
-
-You can't recover such virtual machines through straightforward restarts. The outage could last for several hours and require manual intervention from the Microsoft support team.
-
-##### Is this issue rectified in the latest HDInsight images?
-
-Yes. We fixed this issue on HDInsight images dated on or after March 1, 2022. We advise that you move to the latest stable version to maintain the service-level agreement (SLA) and service reliability.
-
-##### How do I determine the date of the HDInsight image that my clusters are built on?
-
-The last 10 digits in your HDInsight image version indicate the date and time of the image. For example, an image version of 5.0.3000.1.2208310943 indicates a date of August 31, 2022. [Learn how to verify your HDInsight image version](/azure/hdinsight/view-hindsight-cluster-image-version).
-
-#### Resources
-
-- [Create HDInsight clusters by using automation](/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-setup-methods)
-- [Supported HDInsight versions](/azure/hdinsight/hdinsight-component-versioning#supported-hdinsight-versions)
-- [HDInsight Kafka cluster](/azure/hdinsight/kafka/apache-kafka-introduction)
-- [Verify your HDInsight image version](/azure/hdinsight/view-hindsight-cluster-image-version)
 
 ## Recently closed known issues
 
 Select the title to view more information about that specific known issue. Fixed issues are removed after 60 days.
 
-| Issue ID         | Area                   |Title                    | Issue publish date| Status |
+| Issue ID         | Area                   |Title                    | Issue published date| Status |
 |------------------|------------------------|-------------------------|-------------------|-------|
-|Not applicable|Spark|Conda Version Regression in a recent HDInsight release|October 13, 2023|Closed|
+|Not applicable|Spark|[Conda Version Regression in a recent HDInsight release](./hdinsight-known-issues-conda-version-regression.md)|October 13, 2023|Closed|
 
 ## Next steps
 

articles/hdinsight/kafka241-validation-error-arm-templates.md

@@ -0,0 +1,23 @@
+---
+title: Kafka 2.4.1 validation error in Azure Resource Manager templates
+description: Kafka 2.4.1 validation error in ARM templates Known Issue
+ms.service: hdinsight
+ms.topic: troubleshooting-known-issue
+ms.date: 03/26/2024
+---
+
+# Kafka 2.4.1 validation error in ARM templates
+
+**Issue published date**: October 13, 2023
+
+When you're submitting cluster creation requests by using Azure Resource Manager templates (ARM templates), runbooks, PowerShell, the Azure CLI, and other automation tools, you might receive a "BadRequest" error message if you specify `clusterType = "Kafka"`, `HDI version = "5.0"`, and `Kafka version = "2.4.1"`.
+
+## Troubleshooting steps
+
+When you're using [templates or automation tools](/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-setup-methods) to create HDInsight Kafka clusters, choose `componentVersion = "2.4"`. This value enables you to successfully create a Kafka 2.4.1 cluster in HDInsight 5.0.
+
+## Resources
+
+- [Create HDInsight clusters by using automation](/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-setup-methods)
+- [Supported HDInsight versions](/azure/hdinsight/hdinsight-component-versioning#supported-hdinsight-versions)
+- [HDInsight Kafka cluster](/azure/hdinsight/kafka/apache-kafka-introduction)