Skip to content

Commit 4f782c7

Browse files
committed
Updating note and making freshness updates
1 parent 02d7c21 commit 4f782c7

File tree

1 file changed

+71
-80
lines changed

1 file changed

+71
-80
lines changed

articles/synapse-analytics/spark/apache-spark-azure-create-spark-configuration.md

Lines changed: 71 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -5,141 +5,132 @@ author: jejiang
55
ms.author: jejiang
66
ms.reviewer: whhender
77
ms.service: azure-synapse-analytics
8-
ms.topic: tutorial
8+
ms.topic: how-to
99
ms.subservice: spark
10-
ms.date: 04/21/2022
10+
ms.date: 11/20/2024
1111
ms.custom: references_regions
1212
---
1313

1414
# Manage Apache Spark configuration
1515

16-
In this tutorial, you will learn how to create an Apache Spark configuration for your synapse studio. The created Apache Spark configuration can be managed in a standardized manner and when you create Notebook or Apache spark job definition can select the Apache Spark configuration that you want to use with your Apache Spark pool. When you select it, the details of the configuration are displayed.
16+
In this article, you learn how to create an Apache Spark configuration for your synapse studio. The created Apache Spark configuration can be managed in a standardized manner and when you create Notebook or Apache spark job definition can select the Apache Spark configuration that you want to use with your Apache Spark pool. When you select it, the details of the configuration are displayed.
1717

18-
## Create an Apache Spark Configuration
18+
## Create an Apache Spark Configuration
1919

2020
You can create custom configurations from different entry points, such as from the Apache Spark configuration page of an existing spark pool.
2121

2222
## Create custom configurations in Apache Spark configurations
2323

2424
Follow the steps below to create an Apache Spark Configuration in Synapse Studio.
2525

26-
1. Select **Manage** > **Apache Spark configurations**.
27-
2. Click on **New** button to create a new Apache Spark configuration, or click on **Import** a local .json file to your workspace.
28-
3. **New Apache Spark configuration** page will be opened after you click on **New** button.
29-
4. For **Name**, you can enter your preferred and valid name.
30-
5. For **Description**, you can input some description in it.
31-
6. For **Annotations**, you can add annotations by clicking the **New** button, and also you can delete existing annotations by selecting and clicking **Delete** button.
32-
7. For **Configuration properties**, customize the configuration by clicking **Add** button to add properties. If you do not add a property, Azure Synapse will use the default value when applicable.
33-
34-
![Screenshot that create spark configuration.](./media/apache-spark-azure-log-analytics/create-spark-configuration.png)
35-
36-
8. Click on **Continue** button.
37-
9. Click on **Create** button when the validation succeeded.
38-
10. Publish all
39-
40-
41-
> [!NOTE]
26+
1. Select **Manage** > **Apache Spark configurations**.
27+
1. Select **New** button to create a new Apache Spark configuration, or select **Import** a local .json file to your workspace.
28+
1. **New Apache Spark configuration** page will be opened after you select **New** button.
29+
1. For **Name**, you can enter your preferred and valid name.
30+
1. For **Description**, you can input some description in it.
31+
1. For **Annotations**, you can add annotations by clicking the **New** button, and also you can delete existing annotations by selecting and clicking **Delete** button.
32+
1. For **Configuration properties**, customize the configuration by clicking **Add** button to add properties. If you don't add a property, Azure Synapse will use the default value when applicable.
33+
34+
![Screenshot that create spark configuration.](./media/apache-spark-azure-log-analytics/create-spark-configuration.png)
35+
36+
1. Select **Continue** button.
37+
1. Select **Create** button when the validation succeeded.
38+
1. Publish all
39+
40+
> [!NOTE]
41+
> **Upload Apache Spark configuration** feature has been removed.
4242
>
43-
> **Upload Apache Spark configuration** feature has been removed, but Synapse Studio will keep your previously uploaded configuration.
43+
> Pools using an uploaded configuration need to be updated. [Update your pool's configuration](#create-an-apache-spark-configuration-in-already-existing-apache-spark-pool) by selecting an existing configuration or creating a new configuration in the **Apache Spark configuration** menu for the pool. If no new configuration is selected, jobs for these pools will be run using the default configuration in the Spark system settings.
4444
4545
## Create an Apache Spark Configuration in already existing Apache Spark pool
4646

4747
Follow the steps below to create an Apache Spark configuration in an existing Apache Spark pool.
4848

49-
1. Select an existing Apache Spark pool, and click on action "..." button.
50-
2. Select the **Apache Spark configuration** in the content list.
51-
52-
![Screenshot that apache spark configuration.](./media/apache-spark-azure-create-spark-configuration/create-spark-configuration-by-right-click-on-spark-pool.png)
53-
54-
3. For Apache Spark configuration, you can select an already created configuration from the drop-down list, or click on **+New** to create a new configuration.
55-
56-
* If you click **+New**, the Apache Spark Configuration page will open, and you can create a new configuration by following the steps in [Create custom configurations in Apache Spark configurations](#create-custom-configurations-in-apache-spark-configurations).
57-
* If you select an existing configuration, the configuration details will be displayed at the bottom of the page, you can also click the **Edit** button to edit the existing configuration.
58-
59-
![Screenshot that edit spark configuration.](./media/apache-spark-azure-create-spark-configuration/edit-spark-config.png)
60-
61-
4. Click **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use on this Apache Spark pool.
49+
1. Select an existing Apache Spark pool, and select action "..." button.
50+
1. Select the **Apache Spark configuration** in the content list.
51+
52+
![Screenshot that apache spark configuration.](./media/apache-spark-azure-create-spark-configuration/create-spark-configuration-by-right-click-on-spark-pool.png)
53+
54+
1. For Apache Spark configuration, you can select an already created configuration from the drop-down list, or select **+New** to create a new configuration.
55+
56+
* If you select **+New**, the Apache Spark Configuration page will open, and you can create a new configuration by following the steps in [Create custom configurations in Apache Spark configurations](#create-custom-configurations-in-apache-spark-configurations).
57+
* If you select an existing configuration, the configuration details will be displayed at the bottom of the page, you can also select the **Edit** button to edit the existing configuration.
58+
59+
![Screenshot that edit spark configuration.](./media/apache-spark-azure-create-spark-configuration/edit-spark-config.png)
60+
61+
1. Select **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use on this Apache Spark pool.
6262

6363
![Screenshot that select a configuration.](./media/apache-spark-azure-create-spark-configuration/select-a-configuration.png)
6464

65-
5. Click on **Apply** button to save your action.
66-
65+
1. Select **Apply** button to save your action.
6766

6867
## Create an Apache Spark Configuration in the Notebook's configure session
6968

7069
If you need to use a custom Apache Spark Configuration when creating a Notebook, you can create and configure it in the **configure session** by following the steps below.
7170

72-
1. Create a new/Open an existing Notebook.
73-
2. Open the **Properties** of this notebook.
74-
3. Click on **Configure session** to open the Configure session page.
75-
4. Scroll down the configure session page, for Apache Spark configuration, expand the drop-down menu, you can click on New button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration, if you select an existing configuration, click the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
76-
5. Click **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
71+
1. Create a new/Open an existing Notebook.
72+
1. Open the **Properties** of this notebook.
73+
1. Select **Configure session** to open the Configure session page.
74+
1. Scroll down the configure session page, for Apache Spark configuration, expand the drop-down menu, you can select New button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration, if you select an existing configuration, select the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
75+
1. Select **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
7776

78-
![Screenshot that create configuration in configure session.](./media/apache-spark-azure-create-spark-configuration/create-spark-config-in-configure-session.png)
77+
![Screenshot that create configuration in configure session.](./media/apache-spark-azure-create-spark-configuration/create-spark-config-in-configure-session.png)
7978

8079
## Create an Apache Spark Configuration in Apache Spark job definitions
8180

82-
When you are creating a spark job definition, you need to use Apache Spark configuration, which can be created by following the steps below:
83-
84-
1. Create a new/Open an existing Apache Spark job definitions.
85-
2. For **Apache Spark configuration**, you can click on New button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration in the drop-down menu, if you select an existing configuration, click the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
86-
3. Click **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
81+
When you're creating a spark job definition, you need to use Apache Spark configuration, which can be created by following the steps below:
8782

88-
![Screenshot that create configuration in spark job definitions.](./media/apache-spark-azure-create-spark-configuration/create-spark-config-in-spark-job-definition.png)
83+
1. Create a new/Open an existing Apache Spark job definition.
84+
1. For **Apache Spark configuration**, you can select the **New** button to [create a new configuration](#create-custom-configurations-in-apache-spark-configurations). Or select an existing configuration in the drop-down menu, if you select an existing configuration, select the **Edit** icon to go to the Edit Apache Spark configuration page to edit the configuration.
85+
1. Select **View Configurations** to open the **Select a Configuration** page. All configurations will be displayed on this page. You can select a configuration that you want to use.
8986

87+
![Screenshot that create configuration in spark job definitions.](./media/apache-spark-azure-create-spark-configuration/create-spark-config-in-spark-job-definition.png)
9088

91-
> [!NOTE]
89+
> [!NOTE]
9290
>
9391
> If the Apache Spark configuration in the Notebook and Apache Spark job definition does not do anything special, the default configuration will be used when running the job.
9492
95-
9693
## Import and Export an Apache Spark configuration
9794

9895
You can import .txt/.conf/.json config in three formats and then convert it to artifact and publish it. And can also export to one of these three formats.
9996

100-
- Import .txt/.conf/.json configuration from local.
97+
* Import .txt/.conf/.json configuration from local.
10198

10299
![Screenshot that import config.](./media/apache-spark-azure-create-spark-configuration/import-config.png)
103100

104-
105-
- Export .txt/.conf/.json configuration to local.
101+
* Export .txt/.conf/.json configuration to local.
106102

107103
![Screenshot that export config.](./media/apache-spark-azure-create-spark-configuration/export-config.png)
108104

109-
110105
For .txt config file and .conf config file, you can refer to the following examples:
111106

112-
```txt
107+
```txt
113108

114-
spark.synapse.key1 sample
115-
spark.synapse.key2 true
116-
# spark.synapse.key3 sample2
109+
spark.synapse.key1 sample
110+
spark.synapse.key2 true
111+
# spark.synapse.key3 sample2
117112

118-
```
113+
```
119114

120115
For .json config file, you can refer to the following examples:
121116

122-
```json
123-
{
124-
"configs": {
125-
"spark.synapse.key1": "hello world",
126-
"spark.synapse.key2": "true"
127-
},
128-
"annotations": [
129-
"Sample"
130-
]
131-
}
132-
```
133-
134-
> [!NOTE]
135-
>
117+
```json
118+
{
119+
"configs": {
120+
"spark.synapse.key1": "hello world",
121+
"spark.synapse.key2": "true"
122+
},
123+
"annotations": [
124+
"Sample"
125+
]
126+
}
127+
```
128+
129+
> [!NOTE]
136130
> Synapse Studio will continue to support terraform or bicep-based configuration files.
137131
132+
## Related content
138133

139-
## Next steps
140-
141-
- [Use serverless Apache Spark pool in Synapse Studio](../quickstart-create-apache-spark-pool-studio.md).
142-
- [Run a Spark application in notebook](./apache-spark-development-using-notebooks.md).
143-
- [Create Apache Spark job definition in Azure Studio](./apache-spark-job-definitions.md).
144-
- [Collect Apache Spark applications logs and metrics with Azure Storage account](./azure-synapse-diagnostic-emitters-azure-storage.md).
145-
- [Collect Apache Spark applications logs and metrics with Azure Event Hubs](./azure-synapse-diagnostic-emitters-azure-eventhub.md).
134+
* [Use serverless Apache Spark pool in Synapse Studio](../quickstart-create-apache-spark-pool-studio.md).
135+
* [Run a Spark application in notebook](./apache-spark-development-using-notebooks.md).
136+
* [Collect Apache Spark applications logs and metrics with Azure Storage account](./azure-synapse-diagnostic-emitters-azure-storage.md).

0 commit comments

Comments
 (0)