You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/spark/apache-spark-azure-create-spark-configuration.md
+72-81Lines changed: 72 additions & 81 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,141 +5,132 @@ author: jejiang
5
5
ms.author: jejiang
6
6
ms.reviewer: whhender
7
7
ms.service: azure-synapse-analytics
8
-
ms.topic: tutorial
8
+
ms.topic: how-to
9
9
ms.subservice: spark
10
-
ms.date: 04/21/2022
10
+
ms.date: 11/20/2024
11
11
ms.custom: references_regions
12
12
---
13
13
14
14
# Manage Apache Spark configuration
15
15
16
-
In this tutorial, you will learn how to create an Apache Spark configuration for your synapse studio. The created Apache Spark configuration can be managed in a standardized manner and when you create Notebook or Apache spark job definition can select the Apache Spark configuration that you want to use with your Apache Spark pool. When you select it, the details of the configuration are displayed.
16
+
In this article, you learn how to create an Apache Spark configuration for your synapse studio. The created Apache Spark configuration can be managed in a standardized manner and when you create Notebook or Apache spark job definition can select the Apache Spark configuration that you want to use with your Apache Spark pool. When you select it, the details of the configuration are displayed.
17
17
18
-
## Create an Apache Spark Configuration
18
+
## Create an Apache Spark Configuration
19
19
20
20
You can create custom configurations from different entry points, such as from the Apache Spark configuration page of an existing spark pool.
21
21
22
22
## Create custom configurations in Apache Spark configurations
23
23
24
24
Follow the steps below to create an Apache Spark Configuration in Synapse Studio.
2. Click on **New** button to create a new Apache Spark configuration, or click on**Import** a local .json file to your workspace.
28
-
3.**New Apache Spark configuration** page will be opened after you click on**New** button.
29
-
4. For **Name**, you can enter your preferred and valid name.
30
-
5. For **Description**, you can input some description in it.
31
-
6. For **Annotations**, you can add annotations by clicking the **New** button, and also you can delete existing annotations by selecting and clicking **Delete** button.
32
-
7. For **Configuration properties**, customize the configuration by clicking **Add** button to add properties. If you do not add a property, Azure Synapse will use the default value when applicable.
33
-
34
-
35
-
36
-
8. Click on**Continue** button.
37
-
9. Click on**Create** button when the validation succeeded.
1. Select **New** button to create a new Apache Spark configuration, or select**Import** a local .json file to your workspace.
28
+
1.**New Apache Spark configuration** page will be opened after you select**New** button.
29
+
1. For **Name**, you can enter your preferred and valid name.
30
+
1. For **Description**, you can input some description in it.
31
+
1. For **Annotations**, you can add annotations by clicking the **New** button, and also you can delete existing annotations by selecting and clicking **Delete** button.
32
+
1. For **Configuration properties**, customize the configuration by clicking **Add** button to add properties. If you don't add a property, Azure Synapse will use the default value when applicable.
33
+
34
+
35
+
36
+
1. Select**Continue** button.
37
+
1. Select**Create** button when the validation succeeded.
38
+
1. Publish all.
39
+
40
+
> [!NOTE]
41
+
> **Upload Apache Spark configuration** feature has been removed.
42
42
>
43
-
> **Upload Apache Spark configuration**feature has been removed, but Synapse Studio will keep your previously uploaded configuration.
43
+
> Pools using an uploaded configuration need to be updated. [Update your pool's configuration](#create-an-apache-spark-configuration-in-already-existing-apache-spark-pool) by selecting an existing configuration or creating a new configuration in the **Apache Spark configuration**menu for the pool. If no new configuration is selected, jobs for these pools will be run using the default configuration in the Spark system settings.
44
44
45
45
## Create an Apache Spark Configuration in already existing Apache Spark pool
46
46
47
47
Follow the steps below to create an Apache Spark configuration in an existing Apache Spark pool.
48
48
49
-
1. Select an existing Apache Spark pool, and click on action "..." button.
50
-
2. Select the **Apache Spark configuration** in the content list.
51
-
52
-
53
-
54
-
3. For Apache Spark configuration, you can select an already created configuration from the drop-down list, or click on **+New** to create a new configuration.
55
-
56
-
* If you click **+New**, the Apache Spark Configuration page will open, and you can create a new configuration by following the steps in [Create custom configurations in Apache Spark configurations](#create-custom-configurations-in-apache-spark-configurations).
57
-
* If you select an existing configuration, the configuration details will be displayed at the bottom of the page, you can also click the **Edit** button to edit the existing configuration.
58
-
59
-
60
-
61
-
4. Click **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use on this Apache Spark pool.
62
-
63
-
49
+
1. Select an existing Apache Spark pool, and select action "..." button.
50
+
1. Select the **Apache Spark configuration** in the content list.
64
51
65
-
5. Click on **Apply** button to save your action.
52
+
66
53
54
+
1. For Apache Spark configuration, you can select an already created configuration from the drop-down list, or select **+New** to create a new configuration.
55
+
56
+
* If you select **+New**, the Apache Spark Configuration page will open, and you can create a new configuration by following the steps in [Create custom configurations in Apache Spark configurations](#create-custom-configurations-in-apache-spark-configurations).
57
+
* If you select an existing configuration, the configuration details will be displayed at the bottom of the page, you can also select the **Edit** button to edit the existing configuration.
58
+
59
+
60
+
61
+
1. Select **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use on this Apache Spark pool.
62
+
63
+
64
+
65
+
1. Select **Apply** button to save your action.
67
66
68
67
## Create an Apache Spark Configuration in the Notebook's configure session
69
68
70
69
If you need to use a custom Apache Spark Configuration when creating a Notebook, you can create and configure it in the **configure session** by following the steps below.
71
70
72
-
1. Create a new/Open an existing Notebook.
73
-
2. Open the **Properties** of this notebook.
74
-
3. Click on**Configure session** to open the Configure session page.
75
-
4. Scroll down the configure session page, for Apache Spark configuration, expand the drop-down menu, you can click on New button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration, if you select an existing configuration, click the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
76
-
5. Click**View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
71
+
1. Create a new/Open an existing Notebook.
72
+
1. Open the **Properties** of this notebook.
73
+
1. Select**Configure session** to open the Configure session page.
74
+
1. Scroll down the configure session page, for Apache Spark configuration, expand the drop-down menu, you can select New button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration, if you select an existing configuration, select the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
75
+
1. Select**View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
77
76
78
-
77
+
79
78
80
79
## Create an Apache Spark Configuration in Apache Spark job definitions
81
80
82
-
When you are creating a spark job definition, you need to use Apache Spark configuration, which can be created by following the steps below:
83
-
84
-
1. Create a new/Open an existing Apache Spark job definitions.
85
-
2. For **Apache Spark configuration**, you can click on New button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration in the drop-down menu, if you select an existing configuration, click the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
86
-
3. Click **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
81
+
When you're creating a spark job definition, you need to use Apache Spark configuration, which can be created by following the steps below:
87
82
88
-
83
+
1. Create a new/Open an existing Apache Spark job definition.
84
+
1. For **Apache Spark configuration**, you can select the **New** button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration in the drop-down menu, if you select an existing configuration, select the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
85
+
1. Select **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
89
86
87
+
90
88
91
-
> [!NOTE]
89
+
> [!NOTE]
92
90
>
93
91
> If the Apache Spark configuration in the Notebook and Apache Spark job definition does not do anything special, the default configuration will be used when running the job.
94
92
95
-
96
93
## Import and Export an Apache Spark configuration
97
94
98
95
You can import .txt/.conf/.json config in three formats and then convert it to artifact and publish it. And can also export to one of these three formats.
99
96
100
-
- Import .txt/.conf/.json configuration from local.
97
+
* Import .txt/.conf/.json configuration from local.
101
98
102
99
103
100
104
-
105
-
- Export .txt/.conf/.json configuration to local.
101
+
* Export .txt/.conf/.json configuration to local.
106
102
107
103
108
104
109
-
110
105
For .txt config file and .conf config file, you can refer to the following examples:
111
106
112
-
```txt
107
+
```txt
113
108
114
-
spark.synapse.key1 sample
115
-
spark.synapse.key2 true
116
-
# spark.synapse.key3 sample2
109
+
spark.synapse.key1 sample
110
+
spark.synapse.key2 true
111
+
# spark.synapse.key3 sample2
117
112
118
-
```
113
+
```
119
114
120
115
For .json config file, you can refer to the following examples:
121
116
122
-
```json
123
-
{
124
-
"configs": {
125
-
"spark.synapse.key1": "hello world",
126
-
"spark.synapse.key2": "true"
127
-
},
128
-
"annotations": [
129
-
"Sample"
130
-
]
131
-
}
132
-
```
133
-
134
-
> [!NOTE]
135
-
>
117
+
```json
118
+
{
119
+
"configs": {
120
+
"spark.synapse.key1": "hello world",
121
+
"spark.synapse.key2": "true"
122
+
},
123
+
"annotations": [
124
+
"Sample"
125
+
]
126
+
}
127
+
```
128
+
129
+
> [!NOTE]
136
130
> Synapse Studio will continue to support terraform or bicep-based configuration files.
137
131
132
+
## Related content
138
133
139
-
## Next steps
140
-
141
-
-[Use serverless Apache Spark pool in Synapse Studio](../quickstart-create-apache-spark-pool-studio.md).
142
-
-[Run a Spark application in notebook](./apache-spark-development-using-notebooks.md).
143
-
-[Create Apache Spark job definition in Azure Studio](./apache-spark-job-definitions.md).
144
-
-[Collect Apache Spark applications logs and metrics with Azure Storage account](./azure-synapse-diagnostic-emitters-azure-storage.md).
145
-
-[Collect Apache Spark applications logs and metrics with Azure Event Hubs](./azure-synapse-diagnostic-emitters-azure-eventhub.md).
134
+
*[Use serverless Apache Spark pool in Synapse Studio](../quickstart-create-apache-spark-pool-studio.md).
135
+
*[Run a Spark application in notebook](./apache-spark-development-using-notebooks.md).
136
+
*[Collect Apache Spark applications logs and metrics with Azure Storage account](./azure-synapse-diagnostic-emitters-azure-storage.md).
0 commit comments