You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/machine-learning/how-to-submit-spark-jobs.md
+20-19Lines changed: 20 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -169,20 +169,20 @@ To create a job, a standalone Spark job can be defined as a YAML specification f
169
169
- `standard_e32s_v3`
170
170
- `standard_e64s_v3`
171
171
- `runtime_version` - defines the Spark runtime version. The following Spark runtime versions are currently supported:
172
-
- `3.2`
173
172
- `3.3`
173
+
- `3.4`
174
174
> [!IMPORTANT]
175
175
> Azure Synapse Runtime for Apache Spark: Announcements
176
-
> * Azure Synapse Runtime for Apache Spark 3.2:
177
-
> * EOLA Announcement Date: July 8, 2023
178
-
> * End of Support Date: July 8, 2024. After this date, the runtime will be disabled.
179
-
> * For continued support and optimal performance, we advise migrating to Apache Spark 3.3.
176
+
> * Azure Synapse Runtime for Apache Spark 3.3:
177
+
> * EOLA Announcement Date: July 12, 2024
178
+
> * End of Support Date: March 31, 2025. After this date, the runtime will be disabled.
179
+
> * For continued support and optimal performance, we advise migrating to Apache Spark 3.4.
180
180
181
181
This is an example:
182
182
```yaml
183
183
resources:
184
184
instance_type: standard_e8s_v3
185
-
runtime_version: "3.3"
185
+
runtime_version: "3.4"
186
186
```
187
187
- `compute` - this property defines the name of an attached Synapse Spark pool, as shown in this example:
188
188
```yaml
@@ -263,7 +263,7 @@ identity:
263
263
264
264
resources:
265
265
instance_type: standard_e4s_v3
266
-
runtime_version: "3.3"
266
+
runtime_version: "3.4"
267
267
```
268
268
269
269
> [!NOTE]
@@ -315,14 +315,14 @@ To create a standalone Spark job, use the `azure.ai.ml.spark` function, with the
315
315
- `Standard_E32S_V3`
316
316
- `Standard_E64S_V3`
317
317
- `runtime_version` - a key that defines the Spark runtime version. The following Spark runtime versions are currently supported:
318
-
- `3.2.0`
319
318
- `3.3.0`
319
+
- `3.4.0`
320
320
> [!IMPORTANT]
321
321
> Azure Synapse Runtime for Apache Spark: Announcements
322
-
> * Azure Synapse Runtime for Apache Spark 3.2:
323
-
> * EOLA Announcement Date: July 8, 2023
324
-
> * End of Support Date: July 8, 2024. After this date, the runtime will be disabled.
325
-
> * For continued support and optimal performance, we advise migrating to Apache Spark 3.3.
322
+
> * Azure Synapse Runtime for Apache Spark 3.3:
323
+
> * EOLA Announcement Date: July 12, 2024
324
+
> * End of Support Date: March 31, 2025. After this date, the runtime will be disabled.
325
+
> * For continued support and optimal performance, we advise migrating to Apache Spark 3.4.
326
326
327
327
- `compute`- the name of an attached Synapse Spark pool.
328
328
- `inputs` - the inputs for the Spark job. This parameter should pass a dictionary with mappings of the input data bindings used in the job. This dictionary has these values:
@@ -375,7 +375,7 @@ spark_job = spark(
375
375
executor_instances=2,
376
376
resources={
377
377
"instance_type": "Standard_E8S_V3",
378
-
"runtime_version": "3.3.0",
378
+
"runtime_version": "3.4.0",
379
379
},
380
380
inputs={
381
381
"titanic_data": Input(
@@ -426,10 +426,11 @@ To submit a standalone Spark job using the Azure Machine Learning studio UI:
426
426
2. Select **Spark runtime version**.
427
427
> [!IMPORTANT]
428
428
> Azure Synapse Runtime for Apache Spark: Announcements
429
-
> * Azure Synapse Runtime for Apache Spark 3.2:
430
-
> * EOLA Announcement Date: July 8, 2023
431
-
> * End of Support Date: July 8, 2024. After this date, the runtime will be disabled.
432
-
> * For continued support and optimal performance, we advise migrating to Apache Spark 3.3.
429
+
> * Azure Synapse Runtime for Apache Spark 3.3:
430
+
> * EOLA Announcement Date: July 12, 2024
431
+
> * End of Support Date: March 31, 2025. After this date, the runtime will be disabled.
432
+
> * For continued support and optimal performance, we advise migrating to Apache Spark 3.4.
433
+
433
434
3. If you selected **Attached compute**:
434
435
1. Select an attached Synapse Spark pool from the **Select Azure Machine Learning attached compute** menu.
435
436
4. Select **Next**.
@@ -601,7 +602,7 @@ jobs:
601
602
602
603
resources:
603
604
instance_type: standard_e8s_v3
604
-
runtime_version: "3.3"
605
+
runtime_version: "3.4"
605
606
```
606
607
> [!NOTE]
607
608
> To use an attached Synapse Spark pool, define the `compute` property in the sample YAML specification file shown above, instead of `resources` property.
0 commit comments