You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In this article, you can learn how you can enrich the real time events by joining a stream from Kafka with table on ADLS Gen2 using Flink Streaming. We use Flink Streaming API to join events from HDInsight Kafka with attributes from ADLS Gen2, further we use attributes-joined events to sink into another Kafka topic.
13
+
In this article, you can learn how you can enrich the real time events by joining a stream from Kafka with table on ADLS Gen2 using Flink Streaming. We use Flink Streaming API to join events from HDInsight Kafka with attributes from ADLS Gen2. Further we use attributes-joined events to sink into another Kafka topic.
14
14
15
15
## Prerequisites
16
16
17
17
*[Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)
18
18
*[Kafka cluster on HDInsight](../../hdinsight/kafka/apache-kafka-get-started.md)
19
-
*You're required to ensure the network settings are taken care as described on [Using Kafka on HDInsight](../flink/process-and-consume-data.md); that's to make sure HDInsight on AKS and HDInsight clusters are in the same VNet
19
+
*Ensure the network settings are taken care as described on [Using Kafka on HDInsight](../flink/process-and-consume-data.md) to make sure HDInsight on AKS and HDInsight clusters are in the same VNet
20
20
* For this demonstration, we're using a Window VM as maven project develop environment in the same VNet as HDInsight on AKS
21
21
22
22
## Kafka topic preparation
@@ -45,7 +45,7 @@ We're creating a topic called `user_events`.
45
45
46
46
## Prepare file on ADLS Gen2
47
47
48
-
We are creating a file called `item attributes` in our storage
48
+
We're creating a file called `item attributes` in our storage
49
49
50
50
- The purpose is to read a batch of `item attributes` from a file on ADLS Gen2. Each item has the following fields:
51
51
```
@@ -59,7 +59,7 @@ We are creating a file called `item attributes` in our storage
59
59
60
60
## Develop the Apache Flink job
61
61
62
-
In this step we perform the following activities
62
+
In this step, we perform the following activities
63
63
- Enrich the `user_events` topic from Kafka by joining with `item attributes` from a file on ADLS Gen2.
64
64
- We push the outcome of this step, as an enriched user activity of events into a Kafka topic.
65
65
@@ -259,7 +259,7 @@ public class KafkaJoinGen2Demo {
259
259
}
260
260
```
261
261
262
-
## Package jar and submit to Apache Flink
262
+
## Package jar, and submit to Apache Flink
263
263
264
264
We're submitting the packaged jar to Flink:
265
265
@@ -270,13 +270,13 @@ We're submitting the packaged jar to Flink:
270
270
271
271
### Produce real-time `user_events` topic on Kafka
272
272
273
-
We are able to produce real-time user behavior event `user_events` in Kafka.
273
+
We're able to produce real-time user behavior event `user_events` in Kafka.
274
274
275
275
:::image type="content" source="./media/join-stream-kafka-table-filesystem/step-5-kafka-3-2.png" alt-text="Screenshot showing a real-time user behavior event on Kafka 3.2." border="true" lightbox="./media/join-stream-kafka-table-filesystem/step-5-kafka-3-2.png":::
276
276
277
277
### Consume the `itemAttributes` joining with `user_events` on Kafka
278
278
279
-
We are now using `itemAttributes` on filesystem join user activity events `user_events`.
279
+
We're now using `itemAttributes` on filesystem join user activity events `user_events`.
280
280
281
281
:::image type="content" source="./media/join-stream-kafka-table-filesystem/step-6-kafka-3-2.png" alt-text="Screenshot showing Consume the item attributes-joined user activity events on Kafka 3.2." border="true" lightbox="./media/join-stream-kafka-table-filesystem/step-6-kafka-3-2.png":::
0 commit comments