Skip to content

Commit 8b00c8f

Browse files
authored
Merge pull request #271709 from sreekzz/Flink-Update-Batch-6
Modified this page with new screenshots
2 parents 683dadc + 0244a9e commit 8b00c8f

18 files changed

+325
-455
lines changed

articles/hdinsight-aks/flink/azure-databricks.md

Lines changed: 36 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
---
22
title: Incorporate Apache Flink® DataStream into Azure Databricks Delta Lake Table
3-
description: Learn about incorporate Apache Flink® DataStream into Azure Databricks Delta Lake Table
3+
description: Learn about incorporate Apache Flink® DataStream into Azure Databricks Delta Lake Table.
44
ms.service: hdinsight-aks
55
ms.topic: how-to
6-
ms.date: 10/27/2023
6+
ms.date: 04/10/2024
77
---
88

99
# Incorporate Apache Flink® DataStream into Azure Databricks Delta Lake Tables
@@ -12,9 +12,9 @@ This example shows how to sink stream data in Azure ADLS Gen2 from Apache Flink
1212

1313
## Prerequisites
1414

15-
- [Apache Flink 1.16.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)
15+
- [Apache Flink 1.17.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)
1616
- [Apache Kafka 3.2 on HDInsight](../../hdinsight/kafka/apache-kafka-get-started.md)
17-
- [Azure Databricks](/azure/databricks/getting-started/) in the same VNET as HDInsight on AKS
17+
- [Azure Databricks](/azure/databricks/getting-started/) in the same virtual network as HDInsight on AKS
1818
- [ADLS Gen2](/azure/databricks/getting-started/connect-to-azure-storage/) and Service Principal
1919

2020
## Azure Databricks Auto Loader
@@ -25,9 +25,9 @@ Here are the steps how you can use data from Flink in Azure Databricks delta liv
2525

2626
### Create Apache Kafka® table on Apache Flink® SQL
2727

28-
In this step, you can create Kafka table and ADLS Gen2 on Flink SQL. For the purpose of this document, we are using a airplanes_state_real_time table, you can use any topic of your choice.
28+
In this step, you can create Kafka table and ADLS Gen2 on Flink SQL. In this document, we're using a `airplanes_state_real_time table`. You can use any article of your choice.
2929

30-
You are required to update the broker IPs with your Kafka cluster in the code snippet.
30+
You need to update the broker IPs with your Kafka cluster in the code snippet.
3131

3232
```SQL
3333
CREATE TABLE kafka_airplanes_state_real_time (
@@ -68,34 +68,34 @@ Update the container-name and storage-account-name in the code snippet with your
6868

6969
```SQL
7070
CREATE TABLE adlsgen2_airplanes_state_real_time (
71-
`date` STRING,
72-
`geo_altitude` FLOAT,
73-
`icao24` STRING,
74-
`latitude` FLOAT,
75-
`true_track` FLOAT,
76-
`velocity` FLOAT,
77-
`spi` BOOLEAN,
78-
`origin_country` STRING,
79-
`minute` STRING,
80-
`squawk` STRING,
81-
`sensors` STRING,
82-
`hour` STRING,
83-
`baro_altitude` FLOAT,
84-
`time_position` BIGINT,
85-
`last_contact` BIGINT,
86-
`callsign` STRING,
87-
`event_time` STRING,
88-
`on_ground` BOOLEAN,
89-
`category` STRING,
90-
`vertical_rate` FLOAT,
91-
`position_source` INT,
92-
`current_time` STRING,
93-
`longitude` FLOAT
94-
) WITH (
95-
'connector' = 'filesystem',
96-
'path' = 'abfs://<container-name>@<storage-account-name>/flink/airplanes_state_real_time/',
97-
'format' = 'json'
98-
);
71+
`date` STRING,
72+
`geo_altitude` FLOAT,
73+
`icao24` STRING,
74+
`latitude` FLOAT,
75+
`true_track` FLOAT,
76+
`velocity` FLOAT,
77+
`spi` BOOLEAN,
78+
`origin_country` STRING,
79+
`minute` STRING,
80+
`squawk` STRING,
81+
`sensors` STRING,
82+
`hour` STRING,
83+
`baro_altitude` FLOAT,
84+
`time_position` BIGINT,
85+
`last_contact` BIGINT,
86+
`callsign` STRING,
87+
`event_time` STRING,
88+
`on_ground` BOOLEAN,
89+
`category` STRING,
90+
`vertical_rate` FLOAT,
91+
`position_source` INT,
92+
`current_time` STRING,
93+
`longitude` FLOAT
94+
) WITH (
95+
'connector' = 'filesystem',
96+
'path' = 'abfs://<container-name>@<storage-account-name>.dfs.core.windows.net/data/airplanes_state_real_time/flink/airplanes_state_real_time/',
97+
'format' = 'json'
98+
);
9999
```
100100

101101
Further, you can insert Kafka table into ADLSgen2 table on Flink SQL.
@@ -114,9 +114,9 @@ Further, you can insert Kafka table into ADLSgen2 table on Flink SQL.
114114

115115
ADLS Gen2 provides OAuth 2.0 with your Microsoft Entra application service principal for authentication from an Azure Databricks notebook and then mount into Azure Databricks DBFS.
116116

117-
**Let's get service principle appid, tenant id and secret key.**
117+
**Let's get service principle appid, tenant ID, and secret key.**
118118

119-
:::image type="content" source="media/azure-databricks/service-id.png" alt-text="Screenshot shows get service principle appid, tenant ID and secret key." lightbox="media/azure-databricks/service-id.png":::
119+
:::image type="content" source="media/azure-databricks/service-id.png" alt-text="Screenshot shows get service principle appid, tenant ID, and secret key." lightbox="media/azure-databricks/service-id.png":::
120120

121121
**Grant service principle the Storage Blob Data Owner on Azure portal**
122122

134 KB
Loading
-15.3 KB
Loading
-59.9 KB
Loading
178 KB
Loading
80.2 KB
Loading

0 commit comments

Comments
 (0)