You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Removing all instances of Azure Synapse for Apache Spark 2.4 -3.2 runtime content as per leadership team. Any questions please contact [email protected]
Copy file name to clipboardExpand all lines: articles/synapse-analytics/spark/apache-spark-version-support.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -73,10 +73,10 @@ The patch policy differs based on the [runtime lifecycle stage](./runtime-for-ap
73
73
74
74
## Migration between Apache Spark versions - support
75
75
76
-
This guide provides a structured approach for users looking to upgrade their Azure Synapse Runtime for Apache Spark workloads from versions 2.4, 3.1, 3.2, or 3.3 to [the latest GA version, such as 3.4](./apache-spark-34-runtime.md). Upgrading to the most recent version enables users to benefit from performance enhancements, new features, and improved security measures. It's important to note that transitioning to a higher version may require adjustments to your existing Spark code due to incompatibilities or deprecated features.
76
+
This guide provides a structured approach for users looking to upgrade their Azure Synapse Runtime for Apache Spark workloads to [the latest GA version, such as 3.4](./apache-spark-34-runtime.md). Upgrading to the most recent version enables users to benefit from performance enhancements, new features, and improved security measures. It's important to note that transitioning to a higher version may require adjustments to your existing Spark code due to incompatibilities or deprecated features.
77
77
78
78
### Step 1: Evaluate and plan
79
-
-**Assess Compatibility:** Start with reviewing Apache Spark migration guides to identify any potential incompatibilities, deprecated features, and new APIs between your current Spark version (2.4, 3.1, 3.2, or 3.3) and the target version (for example, 3.4).
79
+
-**Assess Compatibility:** Start with reviewing Apache Spark migration guides to identify any potential incompatibilities, deprecated features, and new APIs between your current Spark version and the target version (for example, 3.4).
80
80
-**Analyze Codebase:** Carefully examine your Spark code to identify the use of deprecated or modified APIs. Pay particular attention to SQL queries and User Defined Functions (UDFs), which may be affected by the upgrade.
81
81
82
82
### Step 2: Create a new Spark pool for testing
@@ -88,15 +88,15 @@ This guide provides a structured approach for users looking to upgrade their Azu
88
88
-**Test in Development Environment:** Test your updated code within a development environment in Azure Synapse, not locally. This step is essential for identifying and fixing any issues before moving to production.
89
89
-**Deploy and Monitor:** After thorough testing and validation in the development environment, deploy your application to the new Spark 3.4 pool. It's critical to monitor the application for any unexpected behaviors. Utilize the monitoring tools available in Azure Synapse to keep track of your Spark applications' performance.
90
90
91
-
**Question:** What steps should be taken in migrating from 2.4 to 3.X?
91
+
**Question:** What steps should be taken migrating to 3.X?
92
92
93
93
**Answer:** Refer to the [Apache Spark migration guide](https://spark.apache.org/docs/latest/sql-migration-guide.html).
94
94
95
95
**Question:** I got an error when I tried to upgrade Spark pool runtime using PowerShell cmdlet when they have attached libraries.
96
96
97
97
**Answer:** Don't use PowerShell cmdlet if you have custom libraries installed in your Synapse workspace. Instead follow these steps:
98
-
1. Recreate Spark Pool 3.3 from the ground up.
99
-
1. Downgrade the current Spark Pool 3.3 to 3.1, remove any packages attached, and then upgrade again to 3.3.
98
+
1. Recreate your Spark Pool from the ground up.
99
+
1. Downgrade the current Spark Pool, remove any packages attached, and then upgrade again to [the latest GA version, such as 3.4](./apache-spark-34-runtime.md)
100
100
101
101
**Question:** Why can't I upgrade to 3.4 without recreating a new Spark pool?
0 commit comments