You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/spark/apache-spark-intellij-tool-plugin.md
+16-5Lines changed: 16 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,17 +12,19 @@ ms.date: 09/04/2019
12
12
13
13
# Tutorial: Use Azure Toolkit for IntelliJ to create Apache Spark applications for HDInsight cluster
14
14
15
-
This tutorial demonstrates how to use the Azure Toolkit for IntelliJ plug-in to develop Apache Spark applications written in [Scala](https://www.scala-lang.org/), and then submit them to an HDInsight Spark cluster directly from the IntelliJ integrated development environment (IDE). You can use the plug-in in a few ways:
15
+
This tutorial demonstrates how to develop Apache Spark applications on Azure HDInsight using the **Azure Toolkit**plug-in for the IntelliJ IDE. [Azure HDInsight](../hdinsight-overview.md) is a managed, open-source analytics service in the cloud that allows you to use open-source frameworks like Hadoop, Apache Spark, Apache Hive, and Apache Kafka.
16
16
17
-
* Develop and submit a Scala Spark application on an HDInsight Spark cluster.
17
+
You can use the **Azure Toolkit** plug-in in a few ways:
18
+
19
+
* Develop and submit a Scala Spark application to an HDInsight Spark cluster.
18
20
* Access your Azure HDInsight Spark cluster resources.
19
21
* Develop and run a Scala Spark application locally.
20
22
21
23
In this tutorial, you learn how to:
22
24
> [!div class="checklist"]
23
25
> * Use the Azure Toolkit for IntelliJ plug-in
24
26
> * Develop Apache Spark applications
25
-
> * Submit application to Azure HDInsight cluster
27
+
> * Submit an application to Azure HDInsight cluster
26
28
27
29
## Prerequisites
28
30
@@ -103,6 +105,7 @@ Perform the following steps to install the Scala plugin:
103
105
104
106
d. The **myApp.scala** file then opens in the main view. Replace the default code with the code found below:
105
107
108
+
```scala
106
109
import org.apache.spark.SparkConf
107
110
import org.apache.spark.SparkContext
108
111
@@ -120,10 +123,12 @@ Perform the following steps to install the Scala plugin:
120
123
}
121
124
122
125
}
126
+
```
123
127
124
128
The code reads the data from HVAC.csv (available on all HDInsight Spark clusters), retrieves the rows that have only one digit in the seventh column in the CSV file, and writes the output to `/HVACOut` under the default storage container for the cluster.
125
129
126
130
## Connect to your HDInsight cluster
131
+
127
132
User can either [sign in to Azure subscription](#sign-in-to-your-azure-subscription), or [link a HDInsight cluster](#link-a-cluster) using Ambari username/password or domain joined credential to connect to your HDInsight cluster.
128
133
129
134
### Sign in to your Azure subscription
@@ -363,22 +368,24 @@ It is convenient for you to foresee the script result by sending some code to th
363
368
## Integrate with HDInsight Identity Broker (HIB)
364
369
365
370
### Connect to your HDInsight ESP cluster with Id Broker (HIB)
371
+
366
372
You can follow the normal steps to sign in to Azure subscription to connect to your HDInsight ESP cluster with Id Broker (HIB). After sign in, you will see the cluster list in Azure Explorer. For more instructions, see [Connect to your HDInsight cluster](#connect-to-your-hdinsight-cluster).
367
373
368
374
### Run a Spark Scala application on an HDInsight ESP cluster with Id Broker (HIB)
375
+
369
376
You can follow the normal steps to submit job to HDInsight ESP cluster with Id Broker (HIB). Refer to [Run a Spark Scala application on an HDInsight Spark cluster](#run-a-spark-scala-application-on-an-hdinsight-spark-cluster) for more instructions.
370
377
371
378
We upload the necessary files to a folder named with your sign in account, and you can see the upload path in the configuration file.
372
379
373
380

374
381
375
382
### Spark console on an HDInsight ESP cluster with Id Broker (HIB)
383
+
376
384
You can run Spark Local Console(Scala) or run Spark Livy Interactive Session Console(Scala) on an HDInsight ESP cluster with Id Broker (HIB). Refer to [Spark Console](#spark-console) for more instructions.
377
385
378
386
> [!NOTE]
379
387
> For the HDInsight ESP cluster with Id Broker (HIB), [link a cluster](#link-a-cluster) and [debug Apache Spark applications remotely](#debug-apache-spark-applications-locally-or-remotely-on-an-hdinsight-cluster) is not supported currently.
380
388
381
-
382
389
## Reader-only role
383
390
384
391
When users submit job to a cluster with reader-only role permission, Ambari credentials is required.
@@ -438,11 +445,15 @@ You can convert the existing Spark Scala applications that you created in Intell
438
445
439
446
2. At the root level is a **module** element like the following:
3. Save the changes. Your application should now be compatible with Azure Toolkit for IntelliJ. You can test it by right-clicking the project name in Project. The pop-up menu now has the option **Submit Spark Application to HDInsight**.
448
459
@@ -467,4 +478,4 @@ If you're not going to continue to use this application, delete the cluster that
467
478
In this tutorial, you learned how to use the Azure Toolkit for IntelliJ plug-in to develop Apache Spark applications written in [Scala](https://www.scala-lang.org/), and then submitted them to an HDInsight Spark cluster directly from the IntelliJ integrated development environment (IDE). Advance to the next article to see how the data you registered in Apache Spark can be pulled into a BI analytics tool such as Power BI.
468
479
469
480
> [!div class="nextstepaction"]
470
-
> [Analyze data using BI tools](apache-spark-use-bi-tools.md)
481
+
> [Analyze Apache Spark data using Power BI](apache-spark-use-bi-tools.md)
0 commit comments