You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/spark/apache-spark-create-cluster-cli.md
+6-8Lines changed: 6 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,25 +7,23 @@ ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
8
ms.topic: quickstart
9
9
ms.date: 02/03/2020
10
-
11
-
#Customer intent: As a developer new to Apache Spark on Azure, I need to see how to create a spark cluster.
10
+
#Customer intent: As a developer new to Apache Spark on Azure, I need to see how to create a Spark cluster.
12
11
---
13
12
14
13
# Quickstart: Create Apache Spark cluster in Azure HDInsight using Azure CLI
15
14
16
-
In this quickstart, you learn how to create an Apache Spark cluster in Azure HDInsight using Azure CLI. Apache Spark enables fast data analytics and cluster computing using in-memory processing. The [Azure command-line interface (CLI)](https://docs.microsoft.com/cli/azure/?view=azure-cli-latest) is Microsoft's cross-platform command-line experience for managing Azure resources.
17
-
18
-
If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.
15
+
In this quickstart, you learn how to create an Apache Spark cluster in Azure HDInsight using the Azure command-line interface (CLI). Azure HDInsight is a managed, full-spectrum, open-source analytics service for enterprises. The Apache Spark framework for HDInsight enables fast data analytics and cluster computing using in-memory processing. The Azure CLI is Microsoft's cross-platform command-line experience for managing Azure resources.
19
16
20
17
## Prerequisites
21
18
22
-
Azure CLI. If you haven't installed the Azure CLI, see [Install the Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli) for steps.
19
+
- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio).
20
+
-[Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli), if you don't want to use Azure Cloud Shell.
1. Sign in to your Azure subscription. If you plan to use Azure Cloud Shell, select **Try it** in the upper-right corner of the code block. Else, enter the command below:
26
+
1. Sign in to your Azure subscription. If you plan to use Azure Cloud Shell, select **Try it** in the upper-right corner of the following code block. Else, enter the following command:
29
27
30
28
```azurecli-interactive
31
29
az login
@@ -138,7 +136,7 @@ az group delete \
138
136
139
137
## Next steps
140
138
141
-
In this quickstart, you learned how to create an Apache Spark cluster in Azure HDInsight using Azure CLI. Advance to the next tutorial to learn how to use an HDInsight Spark cluster to run interactive queries on sample data.
139
+
In this quickstart, you learned how to create an Apache Spark cluster in Azure HDInsight using Azure CLI. Advance to the next tutorial to learn how to use an HDInsight cluster to run interactive queries on sample data.
142
140
143
141
> [!div class="nextstepaction"]
144
142
> [Run interactive queries on Apache Spark](./apache-spark-load-data-run-query.md)
Copy file name to clipboardExpand all lines: articles/hdinsight/spark/apache-spark-jupyter-spark-sql-use-portal.md
+27-19Lines changed: 27 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,48 +1,56 @@
1
1
---
2
2
title: 'Quickstart: Create Spark cluster in HDInsight using Azure portal'
3
-
description: This quickstart shows how to use the Azure portal to create an Apache Spark cluster in Azure HDInsight, and run a Spark SQL.
3
+
description: This quickstart shows how to use the Azure portal to create an Apache Spark cluster in Azure HDInsight, and run a Spark SQL query.
4
4
author: hrasheed-msft
5
5
ms.author: hrasheed
6
6
ms.reviewer: jasonh
7
7
ms.service: hdinsight
8
8
ms.topic: quickstart
9
9
ms.date: 09/27/2019
10
10
ms.custom: mvc
11
-
#Customer intent: As a developer new to Apache Spark on Azure, I need to see how to create a spark cluster and query some data.
11
+
#Customer intent: As a developer new to Apache Spark on Azure, I need to see how to create a Spark cluster and query some data.
12
12
---
13
13
14
14
# Quickstart: Create Apache Spark cluster in Azure HDInsight using Azure portal
15
15
16
-
Learn how to create Apache Spark cluster in Azure HDInsight, and how to run Spark SQL queries against Hive tables. Apache Spark enables fast data analytics and cluster computing using in-memory processing. For information on Spark on HDInsight, see [Overview: Apache Spark on Azure HDInsight](apache-spark-overview.md).
16
+
In this quickstart, you use the Azure portal to create an Apache Spark cluster in Azure HDInsight. You then create a Jupyter notebook, and use it to run Spark SQL queries against Apache Hive tables. Azure HDInsight is a managed, full-spectrum, open-source analytics service for enterprises. The Apache Spark framework for HDInsight enables fast data analytics and cluster computing using in-memory processing. Jupyter notebook lets you interact with your data, combine code with markdown text, and do simple visualizations.
17
17
18
-
In this quickstart, you use the Azure portal to create an HDInsight Spark cluster. The cluster uses Azure Storage Blobs as the cluster storage. For more information on using Data Lake Storage Gen2, see [Quickstart: Set up clusters in HDInsight](../../storage/data-lake-storage/quickstart-create-connect-hdi-cluster.md).
- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio).
23
+
24
+
## Create an Apache Spark cluster in HDInsight
25
+
26
+
You use the Azure portal to create an HDInsight cluster that uses Azure Storage Blobs as the cluster storage. For more information on using Data Lake Storage Gen2, see [Quickstart: Set up clusters in HDInsight](../../storage/data-lake-storage/quickstart-create-connect-hdi-cluster.md).
19
27
20
28
> [!IMPORTANT]
21
29
> Billing for HDInsight clusters is prorated per minute, whether you are using them or not. Be sure to delete your cluster after you have finished using it. For more information, see the [Clean up resources](#clean-up-resources) section of this article.
22
30
23
-
If you don't have an Azure subscription, [create a free account](https://azure.microsoft.com/free/) before you begin.
31
+
1. In the Azure portal, select **Create a resource**.
24
32
25
-
## Create an HDInsight Spark cluster
33
+

26
34
27
-
1.In the Azure portal, select **Create a resource**>**Analytics** > **HDInsight**.
35
+
1.On the **New**page, select**Analytics** > **HDInsight**.
28
36
29
-

37
+

30
38
31
39
1. Under **Basics**, provide the following values:
32
40
33
41
|Property |Description |
34
42
|---------|---------|
35
43
|Subscription | From the drop-down, select an Azure subscription used for this cluster. The subscription used for this quickstart is **Azure**. |
36
44
|Resource group | Specify whether you want to create a new resource group or use an existing one. A resource group is a container that holds related resources for an Azure solution. The resource group name used for this quickstart is **myResourceGroup**. |
37
-
|Cluster name | Give a name to your HDInsight Spark cluster. The cluster name used for this quickstart is **myspark2019**.|
45
+
|Cluster name | Give a name to your HDInsight cluster. The cluster name used for this quickstart is **myspark2019**.|
38
46
|Location | Select a location for the resource group. The template uses this location for creating the cluster as well as for the default cluster storage. The location used for this quickstart is **East US**. |
39
47
|Cluster type| Select **Spark** as the cluster type.|
40
48
|Cluster version|This field will auto-populate with the default version once the cluster type has been selected.|
41
49
|Cluster login username| Enter the cluster login username. The default name is *admin*. You use this account to login in to the Jupyter notebook later in the quickstart. |
42
50
|Cluster login password| Enter the cluster login password. |
43
51
|Secure Shell (SSH) username| Enter the SSH username. The SSH username used for this quickstart is **sshuser**. By default, this account shares the same password as the *Cluster Login username* account. |
44
52
45
-

53
+

46
54
47
55
Select **Next: Storage >>** to continue to the **Storage** page.
48
56
@@ -55,13 +63,13 @@ If you don't have an Azure subscription, [create a free account](https://azure.m
55
63
|Primary storage account|Use the auto-populated value.|
56
64
|Container|Use the auto-populated value.|
57
65
58
-

66
+

59
67
60
68
Select **Review + create** to continue.
61
69
62
70
1. Under **Review + create**, select **Create**. It takes about 20 minutes to create the cluster. The cluster must be created before you can proceed to the next session.
63
71
64
-
If you run into an issue with creating HDInsight clusters, it could be that you do not have the right permissions to do so. For more information, see [Access control requirements](../hdinsight-hadoop-create-linux-clusters-portal.md).
72
+
If you run into an issue with creating HDInsight clusters, it could be that you don't have the right permissions to do so. For more information, see [Access control requirements](../hdinsight-hadoop-create-linux-clusters-portal.md).
65
73
66
74
## Create a Jupyter notebook
67
75
@@ -83,13 +91,13 @@ Jupyter Notebook is an interactive notebook environment that supports various pr
83
91
84
92
A new notebook is created and opened with the name Untitled(Untitled.pynb).
85
93
86
-
## Run Spark SQL statements
94
+
## Run Apache Spark SQL statements
87
95
88
96
SQL (Structured Query Language) is the most common and widely used language for querying and defining data. Spark SQL functions as an extension to Apache Spark for processing structured data, using the familiar SQL syntax.
89
97
90
98
1. Verify the kernel is ready. The kernel is ready when you see a hollow circle next to the kernel name in the notebook. Solid circle denotes that the kernel is busy.
91
99
92
-

100
+

93
101
94
102
When you start the notebook for the first time, the kernel performs some tasks in the background. Wait for the kernel to be ready.
95
103
@@ -100,9 +108,9 @@ SQL (Structured Query Language) is the most common and widely used language for
100
108
SHOW TABLES
101
109
```
102
110
103
-
When you use a Jupyter Notebook with your HDInsight Spark cluster, you get a preset `sqlContext` that you can use to run Hive queries using Spark SQL. `%%sql` tells Jupyter Notebook to use the preset `sqlContext` to run the Hive query. The query retrieves the top 10 rows from a Hive table (**hivesampletable**) that comes with all HDInsight clusters by default. It takes about 30 seconds to get the results. The output looks like:
111
+
When you use a Jupyter Notebook with your HDInsight cluster, you get a preset `sqlContext` that you can use to run Hive queries using Spark SQL. `%%sql` tells Jupyter Notebook to use the preset `sqlContext` to run the Hive query. The query retrieves the top 10 rows from a Hive table (**hivesampletable**) that comes with all HDInsight clusters by default. It takes about 30 seconds to get the results. The output looks like:
104
112
105
-

113
+

106
114
107
115
Every time you run a query in Jupyter, your web browser window title shows a **(Busy)** status along with the notebook title. You also see a solid circle next to the **PySpark** text in the top-right corner.
108
116
@@ -115,7 +123,7 @@ SQL (Structured Query Language) is the most common and widely used language for
115
123
116
124
The screen shall refresh to show the query output.
117
125
118
-

126
+

119
127
120
128
1. From the **File** menu on the notebook, select **Close and Halt**. Shutting down the notebook releases the cluster resources.
121
129
@@ -127,11 +135,11 @@ Switch back to the Azure portal, and select **Delete**.
127
135
128
136

129
137
130
-
You can also select the resource group name to open the resource group page, and then select **Delete resource group**. By deleting the resource group, you delete both the HDInsight Spark cluster, and the default storage account.
138
+
You can also select the resource group name to open the resource group page, and then select **Delete resource group**. By deleting the resource group, you delete both the HDInsight cluster, and the default storage account.
131
139
132
140
## Next steps
133
141
134
-
In this quickstart, you learned how to create an HDInsight Spark cluster and run a basic Spark SQL query. Advance to the next tutorial to learn how to use an HDInsight Spark cluster to run interactive queries on sample data.
142
+
In this quickstart, you learned how to create an Apache Spark cluster in HDInsight and run a basic Spark SQL query. Advance to the next tutorial to learn how to use an HDInsight cluster to run interactive queries on sample data.
135
143
136
144
> [!div class="nextstepaction"]
137
145
> [Run interactive queries on Apache Spark](./apache-spark-load-data-run-query.md)
0 commit comments