Merge pull request #252939 from ekote/unsupported24

AnnaMHuff · web-flow · commit f0a8e2802d7f · 2023-09-28T21:05:34.000-06:00
Spark 2.4 EOL - runtime is not supported as of Sep 29
diff --git a/articles/synapse-analytics/machine-learning/setup-environment-cognitive-services.md b/articles/synapse-analytics/machine-learning/setup-environment-cognitive-services.md
@@ -92,7 +92,7 @@ To get started on Azure Kubernetes Service, follow these steps:
 
 1. [Deploy an Azure Kubernetes Service (AKS) cluster using the Azure portal](../../aks/learn/quick-kubernetes-deploy-portal.md)
 
-1. [Install the Apache Spark 2.4.0 helm chart](https://hub.helm.sh/charts/microsoft/spark)
+1. [Install the Apache Spark 2.4.0 helm chart](https://hub.helm.sh/charts/microsoft/spark) - warning: [Spark 2.4](../spark/apache-spark-24-runtime.md) is retired and out of the support.
 
 1. [Install an Azure AI container using Helm](../../ai-services/computer-vision/deploy-computer-vision-on-premises.md)
 
diff --git a/articles/synapse-analytics/overview-faq.yml b/articles/synapse-analytics/overview-faq.yml
@@ -138,7 +138,7 @@ sections:
       - question: |
           What versions of Spark are available?
         answer: |
-          As of May 2021, Azure Synapse Apache Spark fully supports Spark 2.4 and Spark 3.1. As of April 2022, Spark 3.2 is in preview. For a full list of core components and currently supported versions see [Apache Spark version support](./spark/apache-spark-version-support.md).
+          As of September 2023, Azure Synapse Apache Spark fully supports Spark 3.3. For a full list of core components and currently supported versions see [Apache Spark version support](./spark/apache-spark-version-support.md).
           
       - question: |
           Is there an equivalent of DButils in Azure Synapse Spark?
diff --git a/articles/synapse-analytics/spark/apache-spark-24-runtime.md b/articles/synapse-analytics/spark/apache-spark-24-runtime.md
@@ -1,7 +1,7 @@
 ---
-title: Azure Synapse Runtime for Apache Spark 2.4 (EOLA)
-description: Supported versions of Spark, Scala, Python, and .NET for Apache Spark 2.4.
-author: eskot 
+title: Azure Synapse Runtime for Apache Spark 2.4 (unsupported)
+description: Versions of Spark, Scala, Python, and .NET for Apache Spark 2.4.
+author: ekote 
 ms.service: synapse-analytics 
 ms.topic: reference
 ms.subservice: spark
@@ -10,14 +10,17 @@ ms.author: eskot
 ms.custom: has-adal-ref, devx-track-dotnet, devx-track-extended-java, devx-track-python
 ---
 
-# Azure Synapse Runtime for Apache Spark 2.4 (EOLA)
+# Azure Synapse Runtime for Apache Spark 2.4 (unsupported)
 
 Azure Synapse Analytics supports multiple runtimes for Apache Spark. This document will cover the runtime components and versions for the Azure Synapse Runtime for Apache Spark 2.4.
 
-> [!IMPORTANT]
-> * End of life announced (EOLA) for Azure Synapse Runtime for Apache Spark 2.4 has been announced July 29, 2022.
-> * In accordance with the Synapse runtime for Apache Spark lifecycle policy, Azure Synapse runtime for Apache Spark 2.4 will be retired and disabled as of September 29, 2023. After the EOL date, the retired runtimes are unavailable for new Spark pools and existing workflows can't execute. Metadata will temporarily remain in the Synapse workspace.
-> * We recommend that you upgrade your Apache Spark 2.4 workloads to version 3.3 at your earliest convenience. 
+> [!WARNING]
+> End of Support Notification for Azure Synapse Runtime for Apache Spark 2.4
+> * Effective September 29, 2023, the Azure Synapse will discontinue official support for Spark 2.4 Runtimes. 
+> * Post September 29, we will not be addressing any support tickets related to Spark 2.4. There will be no release pipeline in place for bug or security fixes for Spark 2.4. Utilizing Spark 2.4 post the support cutoff date is undertaken at one's own risk. We strongly discourage its continued use due to potential security and functionality concerns.
+> * Recognizing that certain customers may need additional time to transition to a higher runtime version, we are temporarily extending the usage option for Spark 2.4, but we will not provide any official support for it.
+> * We strongly advise to proactively upgrade their workloads to a more recent version of the runtime (e.g., [Azure Synapse Runtime for Apache Spark 3.3 (GA)](./apache-spark-33-runtime.md)).
+
 
 ## Component versions
 |  Component   | Version   |  
diff --git a/articles/synapse-analytics/spark/apache-spark-external-metastore.md b/articles/synapse-analytics/spark/apache-spark-external-metastore.md
@@ -16,7 +16,7 @@ Azure Synapse Analytics allows Apache Spark pools in the same workspace to share
 
 ## Supported Hive Metastore versions
 
-The feature works with both Spark 2.4 and Spark 3.1. The following table shows the supported Hive Metastore versions for each Spark version.
+The feature works with Spark 3.1. The following table shows the supported Hive Metastore versions for each Spark version.
 
 |Spark Version|HMS 0.13.X|HMS 1.2.X|HMS 2.1.X|HMS 2.3.x|HMS 3.1.X|
 |--|--|--|--|--|--|
@@ -161,7 +161,7 @@ If the underlying data of your Hive tables are stored in Azure Blob storage acco
 3. Provide **Name** of the linked service. Record the name of the linked service, this info will be used in Spark configuration shortly.
 4. Select the Azure Blob Storage account. Make sure Authentication method is **Account key**. Currently Spark pool can only access Blob Storage account via account key.
 5. **Test connection** and click **Create**.
-6. After creating the linked service to Blob Storage account, when you run Spark queries, make sure you run below Spark code in the notebook to get access to the the Blob Storage account for the Spark session. Learn more about why you need to do this [here](./apache-spark-secure-credentials-with-tokenlibrary.md).
+6. After creating the linked service to Blob Storage account, when you run Spark queries, make sure you run below Spark code in the notebook to get access to the Blob Storage account for the Spark session. Learn more about why you need to do this [here](./apache-spark-secure-credentials-with-tokenlibrary.md).
 
 ```python
 %%pyspark
@@ -190,7 +190,7 @@ After setting up storage connections, you can query the existing tables in the H
 No credentials found for account xxxxx.blob.core.windows.net in the configuration, and its container xxxxx is not accessible using anonymous credentials. Please check if the container exists first. If it is not publicly available, you have to provide account credentials.
 ```
 
-When use key authentication to your storage account via linked service, you need to take an extra step to get the token for Spark session. Run below code to configure your Spark session before running the query. Learn more about why you need to do this here.
+When using key authentication to your storage account via linked service, you need to take an extra step to get the token for Spark session. Run below code to configure your Spark session before running the query. Learn more about why you need to do this here.
 
 ```python
 %%pyspark
@@ -254,4 +254,4 @@ You can easily fix this issue by appending `/usr/hdp/current/hadoop-client/*` to
 ```text
 Eg: 
 spark.sql.hive.metastore.jars":"/opt/hive-metastore/lib-2.3/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-client/*
-```
+```
diff --git a/articles/synapse-analytics/spark/apache-spark-intelligent-cache-concept.md b/articles/synapse-analytics/spark/apache-spark-intelligent-cache-concept.md
@@ -93,7 +93,7 @@ You won't see the benefit of this feature if:
 
 * Your workload requires large amounts of shuffle, then disabling the Intelligent Cache will free up available space to prevent your job from failing due to insufficient storage space.  
 
-* You're using a Spark 2.4 pool, you'll need to upgrade your pool to the latest version of Spark. 
+* You're using a Spark 3.1 pool, you'll need to upgrade your pool to the latest version of Spark. 
 
 
 ## Learn more
diff --git a/articles/synapse-analytics/spark/apache-spark-performance-hyperspace.md b/articles/synapse-analytics/spark/apache-spark-performance-hyperspace.md
@@ -31,7 +31,7 @@ This document is also available in notebook form, for [Python](https://github.co
 ## Setup
 
 >[!Note]
-> Hyperspace is supported in Azure Synapse Runtime for Apache Spark 2.4 (EOLA), Azure Synapse Runtime for Apache Spark 3.1 (EOLA), and Azure Synapse Runtime for Apache Spark 3.2 (EOLA). However, it should be noted that Hyperspace is not supported in Azure Synapse Runtime for Apache Spark 3.3 (GA).
+> Hyperspace is supported in Azure Synapse Runtime for Apache Spark 3.1 (EOLA), and Azure Synapse Runtime for Apache Spark 3.2 (EOLA). However, it should be noted that Hyperspace is not supported in Azure Synapse Runtime for Apache Spark 3.3 (GA).
 
 To begin with, start a new Spark session. Since this document is a tutorial merely to illustrate what Hyperspace can offer, you will make a configuration change that allows us to highlight what Hyperspace is doing on small datasets. 
 
diff --git a/articles/synapse-analytics/spark/apache-spark-version-support.md b/articles/synapse-analytics/spark/apache-spark-version-support.md
@@ -1,7 +1,7 @@
 ---
 title: Apache Spark version support
 description: Supported versions of Spark, Scala, Python, .NET
-author: eskot
+author: ekote
 ms.service: synapse-analytics 
 ms.topic: reference
 ms.subservice: spark
@@ -13,14 +13,39 @@ ms.reviewer: eskot
 
 # Azure Synapse runtimes
 
-Apache Spark pools in Azure Synapse use runtimes to tie together essential component versions such as Azure Synapse optimizations, packages, and connectors with a specific Apache Spark version. Each runtime will be upgraded periodically to include new improvements, features, and patches.
-
-When you create a serverless Apache Spark pool, you will have the option to select the corresponding Apache Spark version. Based on this, the pool will come pre-installed with the associated runtime components and packages. The runtimes have the following advantages:
-
+Apache Spark pools in Azure Synapse use runtimes to tie together essential component versions such as Azure Synapse optimizations, packages, and connectors with a specific Apache Spark version. Each runtime will be upgraded periodically to include new improvements, features, and patches. When you create a serverless Apache Spark pool, you will have the option to select the corresponding Apache Spark version. Based on this, the pool will come pre-installed with the associated runtime components and packages. The runtimes have the following advantages:
 - Faster session startup times
 - Tested compatibility with specific Apache Spark versions
 - Access to popular, compatible connectors and open-source packages
 
+
+## Supported Azure Synapse runtime releases 
+
+> [!WARNING]
+> End of Support Notification for Azure Synapse Runtime for Apache Spark 2.4
+> * Effective September 29, 2023, the Azure Synapse will discontinue official support for Spark 2.4 Runtimes. 
+> * Post September 29, we will not be addressing any support tickets related to Spark 2.4. There will be no release pipeline in place for bug or security fixes for Spark 2.4. Utilizing Spark 2.4 post the support cutoff date is undertaken at one's own risk. We strongly discourage its continued use due to potential security and functionality concerns.
+> * Recognizing that certain customers may need additional time to transition to a higher runtime version, we are temporarily extending the usage option for Spark 2.4, but we will not provide any official support for it.
+> * We strongly advise to proactively upgrade their workloads to a more recent version of the runtime (e.g., [Azure Synapse Runtime for Apache Spark 3.3 (GA)](./apache-spark-33-runtime.md)).
+
+The following table lists the runtime name, Apache Spark version, and release date for supported Azure Synapse Runtime releases.
+
+| Runtime name                                                               | Release date      | Release stage                   | End of life announcement date | End of life effective date |
+|----------------------------------------------------------------------------|-------------------|---------------------------------|-------------------------------|----------------------------|
+| [Azure Synapse Runtime for Apache Spark 3.3](./apache-spark-33-runtime.md) | Nov 17, 2022      | GA (as of Feb 23, 2023)         | Nov 17, 2023                  | Nov 17, 2024               |
+| [Azure Synapse Runtime for Apache Spark 3.2](./apache-spark-32-runtime.md) | July 8, 2022      | __End of Life Announced (EOLA)__ | July 8, 2023                  | July 8, 2024               |
+| [Azure Synapse Runtime for Apache Spark 3.1](./apache-spark-3-runtime.md)  | May 26, 2021      | __End of Life Announced (EOLA)__ | January 26, 2023              | January 26, 2024           |
+| [Azure Synapse Runtime for Apache Spark 2.4](./apache-spark-24-runtime.md) | December 15, 2020 | __End of Life (EOL)__           | __July 29, 2022__             | __September 29, 2023__     |
+
+## Runtime release stages
+
+For the complete runtime for Apache Spark lifecycle and support policies, refer to [Synapse runtime for Apache Spark lifecycle and supportability](./runtime-for-apache-spark-lifecycle-and-supportability.md).
+
+## Runtime patching
+
+Azure Synapse runtime for Apache Spark patches are rolled out monthly containing bug, feature and security fixes to the Apache Spark core engine, language environments, connectors and libraries.
+
+
 > [!NOTE]
 > - Maintenance updates will be automatically applied to new sessions for a given serverless Apache Spark pool. 
 > - You should test and validate that your applications run properly when using new runtime versions.
@@ -41,25 +66,6 @@ When you create a serverless Apache Spark pool, you will have the option to sele
 > * ```org/apache/log4j/chainsaw/*```
 >
 > While the above classes were not used in the default Log4j configurations in Synapse, it is possible that some user application could still depend on it. If your application needs to use these classes, use Library Management to add a secure version of Log4j to the Spark Pool. __Do not use Log4j version 1.2.17__, as it would be reintroducing the vulnerabilities.
->
-
-## Supported Azure Synapse runtime releases 
-The following table lists the runtime name, Apache Spark version, and release date for supported Azure Synapse Runtime releases.
-
-| Runtime name                                                               | Release date      | Release stage                    | End of life announcement date | End of life effective date |
-|----------------------------------------------------------------------------|-------------------|----------------------------------|-------------------------------|----------------------------|
-| [Azure Synapse Runtime for Apache Spark 3.3](./apache-spark-33-runtime.md) | Nov 17, 2022      | GA (as of Feb 23, 2023)          | Nov 17, 2023                  | Nov 17, 2024               |
-| [Azure Synapse Runtime for Apache Spark 3.2](./apache-spark-32-runtime.md) | July 8, 2022      | __End of Life Announced (EOLA)__ | July 8, 2023                  | July 8, 2024               |
-| [Azure Synapse Runtime for Apache Spark 3.1](./apache-spark-3-runtime.md)  | May 26, 2021      | __End of Life Announced (EOLA)__ | January 26, 2023              | January 26, 2024           |
-| [Azure Synapse Runtime for Apache Spark 2.4](./apache-spark-24-runtime.md) | December 15, 2020 | __End of Life Announced (EOLA)__ | __July 29, 2022__             | __September 29, 2023__     |
-
-## Runtime release stages
-
-For the complete runtime for Apache Spark lifecycle and support policies, refer to [Synapse runtime for Apache Spark lifecycle and supportability](./runtime-for-apache-spark-lifecycle-and-supportability.md).
-
-## Runtime patching
-
-Azure Synapse runtime for Apache Spark patches are rolled out monthly containing bug, feature and security fixes to the Apache Spark core engine, language environments, connectors and libraries.
 
 The patch policy differs based on the [runtime lifecycle stage](./runtime-for-apache-spark-lifecycle-and-supportability.md):
 1. Generally Available (GA) runtime: Receive no upgrades on major versions (i.e. 3.x -> 4.x). And will upgrade a minor version (i.e. 3.x -> 3.y) as long as there are no deprecation or regression impacts.
diff --git a/articles/synapse-analytics/spark/data-sources/apache-spark-cdm-connector.md b/articles/synapse-analytics/spark/data-sources/apache-spark-cdm-connector.md
@@ -20,7 +20,7 @@ For information on defining Common Data Model documents by using Common Data Mod
 
 At a high level, the connector supports:
 
-* Spark 2.4, 3.1, and 3.2.
+* 3.1, and 3.2., and 3.3.
 * Reading data from an entity in a Common Data Model folder into a Spark DataFrame.
 * Writing from a Spark DataFrame to an entity in a Common Data Model folder based on a Common Data Model entity definition.
 * Writing from a Spark DataFrame to an entity in a Common Data Model folder based on the DataFrame schema.
diff --git a/articles/synapse-analytics/spark/data-sources/apache-spark-kusto-connector.md b/articles/synapse-analytics/spark/data-sources/apache-spark-kusto-connector.md
@@ -12,20 +12,19 @@ author: midesa
 ---
 
 # Azure Data Explorer (Kusto) connector for Apache Spark
-The Azure Data Explorer (Kusto) connector for Apache Spark is designed to efficiently transfer data between Kusto clusters and Spark. This connector is available in Python, Java, and .NET. It is built in to the Azure Synapse Apache Spark 2.4 runtime (EOLA).
+The Azure Data Explorer (Kusto) connector for Apache Spark is designed to efficiently transfer data between Kusto clusters and Spark. This connector is available in Python, Java, and .NET.
 
 ## Authentication
 When using Azure Synapse Notebooks or Apache Spark job definitions, the authentication between systems is made seamless with the linked service. The Token Service connects with Azure Active Directory to obtain security tokens for use when accessing the Kusto cluster.
 
-For Azure Synapse Pipelines, the authentication will use the service principal name. Currently, managed identities are not supported with the Azure Data Explorer connector.
+For Azure Synapse Pipelines, the authentication uses the service principal name. Currently, managed identities aren't supported with the Azure Data Explorer connector.
 
 ## Prerequisites 
-  - [Connect to Azure Data Explorer](../../quickstart-connect-azure-data-explorer.md): You will need to set up a Linked Service to connect to an existing Kusto cluster.
+  - [Connect to Azure Data Explorer](../../quickstart-connect-azure-data-explorer.md): You need to set up a Linked Service to connect to an existing Kusto cluster.
 
 ## Limitations
-  - The Azure Data Explorer (Kusto) connector is currently only supported on the Azure Synapse Apache Spark 2.4 runtime (EOLA).
   - The Azure Data Explorer linked service can only be configured with the Service Principal Name.
-  - Within Azure Synapse Notebooks or Apache Spark Job Definitions, the Azure Data Explorer connector will use Azure AD pass-through to connect to the Kusto Cluster.
+  - Within Azure Synapse Notebooks or Apache Spark Job Definitions, the Azure Data Explorer connector uses Azure AD pass-through to connect to the Kusto Cluster.
 
 
 ## Use the Azure Data Explorer (Kusto) connector
diff --git a/articles/synapse-analytics/spark/low-shuffle-merge-for-apache-spark.md b/articles/synapse-analytics/spark/low-shuffle-merge-for-apache-spark.md
@@ -29,9 +29,9 @@ It's available on Synapse Pools for Apache Spark versions 3.2 and 3.3.
 
 |Version| Availability | Default |
 |--|--|--|
-| Delta 0.6 / Spark 2.4 | No | - |
-| Delta 1.2 / Spark 3.2 | Yes | false |
-| Delta 2.2 / Spark 3.3 | Yes | true |
+| Delta 0.6 / [Spark 2.4](./apache-spark-24-runtime.md) | No | - |
+| Delta 1.2 / [Spark 3.2](./apache-spark-32-runtime.md) | Yes | false |
+| Delta 2.2 / [Spark 3.3](./apache-spark-33-runtime.md) | Yes | true |
 
 
 ## Benefits of Low Shuffle Merge
diff --git a/articles/synapse-analytics/spark/spark-dotnet.md b/articles/synapse-analytics/spark/spark-dotnet.md
@@ -118,7 +118,7 @@ The following features are available when you use .NET for Apache Spark in the A
 
 ### `DotNetRunner: null` / `Futures timeout` in Synapse Spark Job Definition Run
 
-Synapse Spark Job Definitions on Spark Pools using Spark 2.4 require `Microsoft.Spark` 1.0.0. Clear your `bin` and `obj` directories, and publish the project using 1.0.0.
+Synapse Spark Job Definitions on Spark Pools using [Spark 2.4](./apache-spark-24-runtime.md) require `Microsoft.Spark` 1.0.0. Clear your `bin` and `obj` directories, and publish the project using 1.0.0.
 
 ### OutOfMemoryError: java heap space at org.apache.spark
 
diff --git a/articles/synapse-analytics/spark/synapse-spark-sql-pool-import-export.md b/articles/synapse-analytics/spark/synapse-spark-sql-pool-import-export.md
diff --git a/articles/synapse-analytics/sql/resources-self-help-sql-on-demand.md b/articles/synapse-analytics/sql/resources-self-help-sql-on-demand.md
diff --git a/articles/synapse-analytics/toc.yml b/articles/synapse-analytics/toc.yml
