You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/storage/blobs/data-lake-storage-use-databricks-spark.md
+16-52Lines changed: 16 additions & 52 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ author: normesta
7
7
ms.subservice: data-lake-storage-gen2
8
8
ms.service: storage
9
9
ms.topic: tutorial
10
-
ms.date: 11/19/2019
10
+
ms.date: 02/01/2023
11
11
ms.author: normesta
12
12
ms.reviewer: dineshm
13
13
ms.custom: devx-track-python, py-fresh-zinc
@@ -21,7 +21,6 @@ This tutorial shows you how to connect your Azure Databricks cluster to data sto
21
21
In this tutorial, you will:
22
22
23
23
> [!div class="checklist"]
24
-
> - Create a Databricks cluster
25
24
> - Ingest unstructured data into a storage account
26
25
> - Run analytics on your data in Blob storage
27
26
@@ -41,58 +40,18 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
41
40
42
41
See [Tutorial: Connect to Azure Data Lake Storage Gen2](/azure/databricks/getting-started/connect-to-azure-storage) (Steps 1 through 3). After completing these steps, make sure to paste the tenant ID, app ID, and client secret values into a text file. You'll need those soon.
43
42
44
-
### Download the flight data
43
+
- An Azure Databricks workspace. See [Create an Azure Databricks workspace](/azure/databricks/getting-started/#--create-an-azure-databricks-workspace).
44
+
45
+
- An Azure Databricks cluster. See [Create a cluster](/azure/databricks/getting-started/quick-start#step-1-create-a-cluster).
46
+
47
+
## Download the flight data
45
48
46
49
This tutorial uses flight data from the Bureau of Transportation Statistics to demonstrate how to perform an ETL operation. You must download this data to complete the tutorial.
47
50
48
51
1. Download the [On_Time_Reporting_Carrier_On_Time_Performance_1987_present_2016_1.zip](https://github.com/Azure-Samples/AzureStorageSnippets/blob/master/blobs/tutorials/On_Time_Reporting_Carrier_On_Time_Performance_1987_present_2016_1.zip) file. This file contains the flight data.
49
52
50
53
2. Unzip the contents of the zipped file and make a note of the file name and the path of the file. You need this information in a later step.
51
54
52
-
## Create an Azure Databricks service
53
-
54
-
In this section, you create an Azure Databricks service by using the Azure portal.
55
-
56
-
1. In the Azure portal, select **Create a resource** > **Analytics** > **Azure Databricks**.
57
-
58
-

59
-
60
-
2. Under **Azure Databricks Service**, provide the following values to create a Databricks service:
61
-
62
-
|Property |Description |
63
-
|---------|---------|
64
-
|**Workspace name**| Provide a name for your Databricks workspace. |
65
-
|**Subscription**| From the drop-down, select your Azure subscription. |
66
-
|**Resource group**| Specify whether you want to create a new resource group or use an existing one. A resource group is a container that holds related resources for an Azure solution. For more information, see [Azure Resource Group overview](../../azure-resource-manager/management/overview.md). |
67
-
|**Location**| Select **West US 2**. For other available regions, see [Azure services available by region](https://azure.microsoft.com/regions/services/). |
68
-
|**Pricing Tier**| Select **Standard**. |
69
-
70
-

71
-
72
-
3. The account creation takes a few minutes. To monitor the operation status, view the progress bar at the top.
73
-
74
-
4. Select **Pin to dashboard** and then select **Create**.
75
-
76
-
## Create a Spark cluster in Azure Databricks
77
-
78
-
1. In the Azure portal, go to the Databricks service that you created, and select **Launch Workspace**.
79
-
80
-
2. You're redirected to the Azure Databricks portal. From the portal, select **Cluster**.
81
-
82
-

83
-
84
-
3. In the **New cluster** page, provide the values to create a cluster.
85
-
86
-

87
-
88
-
Fill in values for the following fields, and accept the default values for the other fields:
89
-
90
-
- Enter a name for the cluster.
91
-
92
-
- Make sure you select the **Terminate after 120 minutes of inactivity** checkbox. Provide a duration (in minutes) to terminate the cluster, if the cluster is not being used.
93
-
94
-
4. Select **Create cluster**. After the cluster is running, you can attach notebooks to the cluster and run Spark jobs.
95
-
96
55
## Ingest data
97
56
98
57
### Copy source data into the storage account
@@ -125,15 +84,20 @@ In this section, you'll create a container and a folder in your storage account.
125
84
126
85
1. In the [Azure portal](https://portal.azure.com), go to the Azure Databricks service that you created, and select **Launch Workspace**.
127
86
128
-
2. On the left, select **Workspace**. From the **Workspace** drop-down, select **Create** > **Notebook**.
87
+
2. In the sidebar, select **Workspace**.
88
+
89
+
3. In the Workspace folder, select **Create > Notebook**.
90
+
91
+
> [!div class="mx-imgBorder"]
92
+
> 
129
93
130
-

94
+
4. In the **Create Notebook** dialog, enter a name and then select **Python**in the **Default Language** drop-down list. This selection determines the default language of the notebook.
131
95
132
-
3. In the **Create Notebook**dialog box, enter a name for the notebook. Select **Python** as the language, and then select the Spark cluster that you created earlier.
96
+
5. In the **Cluster**drop-down list, make sure that the cluster you created earlier is selected.
133
97
134
-
4. Select**Create**.
98
+
6. Click**Create**. The notebook opens with an empty cell at the top.
135
99
136
-
5. Copy and paste the following code block into the first cell, but don't run this code yet.
100
+
7. Copy and paste the following code block into the first cell, but don't run this code yet.
0 commit comments