You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/apache-kafka-spark-structured-streaming-cosmosdb.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ Learn how to use [Apache Spark](https://spark.apache.org/) [Structured Streaming
16
16
Spark structured streaming is a stream processing engine built on Spark SQL. It allows you to express streaming computations the same as batch computation on static data. For more information on Structured Streaming, see the [Structured Streaming Programming Guide](https://spark.apache.org/docs/2.2.0/structured-streaming-programming-guide.html) at Apache.org.
17
17
18
18
> [!IMPORTANT]
19
-
> This example used Spark 2.2 on HDInsight 3.6.
19
+
> This example uses Spark 2.4 on HDInsight 4.0.
20
20
>
21
21
> The steps in this document create an Azure resource group that contains both a Spark on HDInsight and a Kafka on HDInsight cluster. These clusters are both located within an Azure Virtual Network, which allows the Spark cluster to directly communicate with the Kafka cluster.
22
22
>
@@ -43,16 +43,16 @@ While you can create an Azure virtual network, Kafka, and Spark clusters manuall
43
43
44
44
This template creates the following resources:
45
45
46
-
* A Kafka on HDInsight 3.6 cluster.
46
+
* A Kafka on HDInsight 4.0 cluster.
47
47
48
-
* A Spark on HDInsight 3.6 cluster.
48
+
* A Spark on HDInsight 4.0 cluster.
49
49
50
50
* An Azure Virtual Network, which contains the HDInsight clusters. The virtual network created by the template uses the 10.0.0.0/16 address space.
51
51
52
52
* An Azure Cosmos DB SQL API database.
53
53
54
54
> [!IMPORTANT]
55
-
> The structured streaming notebook used in this example requires Spark on HDInsight 3.6. If you use an earlier version of Spark on HDInsight, you receive errors when using the notebook.
55
+
> The structured streaming notebook used in this example requires Spark on HDInsight 4.0. If you use an earlier version of Spark on HDInsight, you receive errors when using the notebook.
56
56
57
57
1. Use the following information to populate the entries on the **Custom deployment** section:
58
58
@@ -62,7 +62,7 @@ While you can create an Azure virtual network, Kafka, and Spark clusters manuall
62
62
|Resource group|Create a group or select an existing one. This group contains the HDInsight cluster.|
63
63
|Cosmos DB Account Name|This value is used as the name for the Cosmos DB account. The name can only contain lowercase letters, numbers, and the hyphen (-) character. It must be between 3-31 characters in length.|
64
64
|Base Cluster Name|This value is used as the base name for the Spark and Kafka clusters. For example, entering **myhdi** creates a Spark cluster named __spark-myhdi__ and a Kafka cluster named **kafka-myhdi**.|
65
-
|Cluster Version|The HDInsight cluster version. This example is tested with HDInsight 3.6, and may not work with other cluster types.|
65
+
|Cluster Version|The HDInsight cluster version. This example is tested with HDInsight 4.0, and may not work with other cluster types.|
66
66
|Cluster Login User Name|The admin user name for the Spark and Kafka clusters.|
67
67
|Cluster Login Password|The admin user password for the Spark and Kafka clusters.|
68
68
|Ssh User Name|The SSH user to create for the Spark and Kafka clusters.|
0 commit comments