Skip to content

Commit 67575d1

Browse files
committed
spark connector and pngs update
1 parent 4ddad2b commit 67575d1

File tree

4 files changed

+24
-25
lines changed

4 files changed

+24
-25
lines changed
38.3 KB
Loading
8.85 KB
Loading
-7.78 KB
Loading

articles/data-explorer/spark-connector.md

Lines changed: 24 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ ms.author: orspodek
66
ms.reviewer: michazag
77
ms.service: data-explorer
88
ms.topic: conceptual
9-
ms.date: 4/29/2019
9+
ms.date: 1/14/2020
1010
---
1111

1212
# Azure Data Explorer Connector for Apache Spark (Preview)
@@ -30,7 +30,7 @@ and sink operations such as write, read and writeStream.
3030
* Install Azure Data Explorer connector library, and libraries listed in [dependencies](https://github.com/Azure/azure-kusto-spark#dependencies) including the following [Kusto Java SDK](/azure/kusto/api/java/kusto-java-client-library) libraries:
3131
* [Kusto Data Client](https://mvnrepository.com/artifact/com.microsoft.azure.kusto/kusto-data)
3232
* [Kusto Ingest Client](https://mvnrepository.com/artifact/com.microsoft.azure.kusto/kusto-ingest)
33-
* Pre-built libraries for [Spark 2.4, Scala 2.11](https://github.com/Azure/azure-kusto-spark/releases)
33+
* Pre-built libraries for [Spark 2.4, Scala 2.11](https://github.com/Azure/azure-kusto-spark/releases) and [Maven repo](https://mvnrepository.com/artifact/com.microsoft.azure.kusto/spark-kusto-connector)
3434

3535
## How to build the Spark connector
3636

@@ -79,24 +79,17 @@ For more information, see [connector usage](https://github.com/Azure/azure-kusto
7979
> [!NOTE]
8080
> It is recommended to use the latest Azure Data Explorer Spark connector release when performing the following steps:
8181
82-
1. Set the following Spark cluster settings, based on Azure Databricks cluster using Spark 2.4 and Scala 2.11:
82+
1. Set the following Spark cluster settings, based on Azure Databricks cluster using Spark 2.4.4 and Scala 2.11:
8383

84-
![Databricks cluster settings](media/spark-connector/databricks-cluster.png)
84+
[Databricks cluster settings](media/spark-connector/databricks-cluster.png)
8585

86-
1. Import the Azure Data Explorer connector library:
86+
1. Install the latest spark-kusto-connector library from Maven::
8787

88-
![Import Azure Data Explorer library](media/spark-connector/db-create-library.png)
89-
90-
1. Add additional dependencies (not necessary if used from maven) :
91-
92-
![Add dependencies](media/spark-connector/db-dependencies.png)
93-
94-
> [!TIP]
95-
> The correct java release version for each Spark release is found [here](https://github.com/Azure/azure-kusto-spark#dependencies).
88+
[Import Azure Data Explorer library](media/spark-connector/db-create-library.png)
9689

9790
1. Verify that all required libraries are installed:
9891

99-
![Verify libraries installed](media/spark-connector/db-libraries-view.png)
92+
[Verify libraries installed](media/spark-connector/db-libraries-view.png)
10093

10194
## Authentication
10295

@@ -141,19 +134,19 @@ For more information on Azure Data Explorer principal roles, see [role-based aut
141134

142135
```scala
143136
import com.microsoft.kusto.spark.datasink.KustoSinkOptions
144-
val conf = Map(
145-
KustoSinkOptions.KUSTO_CLUSTER -> cluster,
146-
KustoSinkOptions.KUSTO_TABLE -> table,
147-
KustoSinkOptions.KUSTO_DATABASE -> database,
148-
KustoSinkOptions.KUSTO_AAD_CLIENT_ID -> appId,
149-
KustoSinkOptions.KUSTO_AAD_CLIENT_PASSWORD -> appKey,
150-
KustoSinkOptions.KUSTO_AAD_AUTHORITY_ID -> authorityId)
151-
137+
import org.apache.spark.sql.{SaveMode, SparkSession}
138+
152139
df.write
153140
.format("com.microsoft.kusto.spark.datasource")
154-
.options(conf)
155-
.save()
156-
141+
.option(KustoSinkOptions.KUSTO_CLUSTER, cluster)
142+
.option(KustoSinkOptions.KUSTO_DATABASE, database)
143+
.option(KustoSinkOptions.KUSTO_TABLE, "Demo3_spark")
144+
.option(KustoSinkOptions.KUSTO_AAD_CLIENT_ID, appId)
145+
.option(KustoSinkOptions.KUSTO_AAD_CLIENT_PASSWORD, appKey)
146+
.option(KustoSinkOptions.KUSTO_AAD_AUTHORITY_ID, authorityId)
147+
.option(KustoSinkOptions.KUSTO_TABLE_CREATE_OPTIONS, "CreateIfNotExist")
148+
.mode(SaveMode.Append)
149+
.save()
157150
```
158151

159152
Or use the simplified syntax:
@@ -249,3 +242,9 @@ For more information on Azure Data Explorer principal roles, see [role-based aut
249242

250243
display(dfFiltered)
251244
```
245+
246+
## Next steps
247+
248+
* Learn more about the [Azure Data Explorer Spark Connector](https://github.com/Azure/azure-kusto-spark/tree/master/docs)
249+
* [Sample code](https://github.com/Azure/azure-kusto-spark/tree/master/samples/src/main)
250+

0 commit comments

Comments
 (0)