Skip to content

Commit 5a114be

Browse files
ttorblettorble
authored
Merge pull request #270879 from sreekzz/Flink-batch3
Replaced Images for Flink 1.17
2 parents 0cacd08 + 39bbe0d commit 5a114be

File tree

5 files changed

+109
-110
lines changed

5 files changed

+109
-110
lines changed

‎articles/hdinsight-aks/flink/cosmos-db-for-apache-cassandra.md

Lines changed: 106 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -3,20 +3,20 @@ title: Using Azure Cosmos DB for Apache Cassandra® with HDInsight on AKS for Ap
33
description: Learn how to Sink Apache Kafka® message into Azure Cosmos DB for Apache Cassandra®, with Apache Flink® running on HDInsight on AKS.
44
ms.service: hdinsight-aks
55
ms.topic: how-to
6-
ms.date: 10/30/2023
6+
ms.date: 04/02/2024
77
---
88

99
# Sink Apache Kafka® messages into Azure Cosmos DB for Apache Cassandra, with Apache Flink® on HDInsight on AKS
1010

1111
[!INCLUDE [feature-in-preview](../includes/feature-in-preview.md)]
1212

13-
This example uses [Apache Flink](../flink/flink-overview.md) to sink [HDInsight for Apache Kafka](/azure/hdinsight/kafka/apache-kafka-introduction) messages into [Azure Cosmos DB for Apache Cassandra](/azure/cosmos-db/cassandra/introduction)
13+
This example uses [Apache Flink](../flink/flink-overview.md) to sink [HDInsight for Apache Kafka](/azure/hdinsight/kafka/apache-kafka-introduction) messages into [Azure Cosmos DB for Apache Cassandra](/azure/cosmos-db/cassandra/introduction).
1414

1515
This example is prominent when Engineers prefer real-time aggregated data for analysis. With access to historical aggregated data, you can build machine learning (ML) models to build insights or actions. You can also ingest IoT data into Apache Flink to aggregate data in real-time and store it in Apache Cassandra.
1616

1717
## Prerequisites
1818

19-
* [Apache Flink 1.16.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)
19+
* [Apache Flink 1.17.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)
2020
* [Apache Kafka 3.2 on HDInsight](../../hdinsight/kafka/apache-kafka-get-started.md)
2121
* [Azure Cosmos DB for Apache Cassandra](../../cosmos-db/cassandra/index.yml)
2222
* An Ubuntu VM for maven project development environment in the same VNet as HDInsight on AKS cluster.
@@ -42,115 +42,114 @@ Get credentials uses it on Stream source code:
4242

4343
### Cloning repository of Azure Samples
4444

45-
Refer GitHub readme to download maven, clone this repository using `Azure-Samples/azure-cosmos-db-cassandra-java-getting-started.git` from
46-
[Azure Samples ](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started)
45+
Refer GitHub readme to download maven, and clone this repository using `Azure-Samples/azure-cosmos-db-cassandra-java-getting-started.git` from
46+
[Azure Samples ](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started).
4747

4848
### Updating maven project for Cassandra
4949

50-
Go to maven project folder **azure-cosmos-db-cassandra-java-getting-started-main** and update the changes required for this example
50+
Go to maven project folder **azure-cosmos-db-cassandra-java-getting-started-main** and update the changes required for this example.
5151

5252
**maven pom.xml**
5353
``` xml
54-
5554
<?xml version="1.0" encoding="UTF-8"?>
5655
<project xmlns="http://maven.apache.org/POM/4.0.0"
57-
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
58-
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
59-
<modelVersion>4.0.0</modelVersion>
60-
61-
<groupId>com.azure.cosmosdb.cassandra</groupId>
62-
<artifactId>cosmosdb-cassandra-examples</artifactId>
63-
<version>1.0-SNAPSHOT</version>
64-
<dependencies>
65-
<dependency>
66-
<groupId>org.apache.flink</groupId>
67-
<artifactId>flink-java</artifactId>
68-
<version>1.16.0</version>
69-
</dependency>
70-
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java -->
71-
<dependency>
72-
<groupId>org.apache.flink</groupId>
73-
<artifactId>flink-streaming-java</artifactId>
74-
<version>1.16.0</version>
75-
</dependency>
76-
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
77-
<dependency>
78-
<groupId>org.apache.flink</groupId>
79-
<artifactId>flink-clients</artifactId>
80-
<version>1.16.0</version>
81-
</dependency>
82-
<dependency>
83-
<groupId>org.apache.flink</groupId>
84-
<artifactId>flink-connector-files</artifactId>
85-
<version>1.16.0</version>
86-
</dependency>
87-
<dependency>
88-
<groupId>org.apache.flink</groupId>
89-
<artifactId>flink-connector-kafka</artifactId>
90-
<version>1.16.0</version>
91-
</dependency>
92-
<dependency>
93-
<groupId>com.datastax.cassandra</groupId>
94-
<artifactId>cassandra-driver-core</artifactId>
95-
<version>3.3.0</version>
96-
</dependency>
97-
<dependency>
98-
<groupId>com.datastax.cassandra</groupId>
99-
<artifactId>cassandra-driver-mapping</artifactId>
100-
<version>3.1.4</version>
101-
</dependency>
102-
<dependency>
103-
<groupId>com.datastax.cassandra</groupId>
104-
<artifactId>cassandra-driver-extras</artifactId>
105-
<version>3.1.4</version>
106-
</dependency>
107-
<dependency>
108-
<groupId>org.slf4j</groupId>
109-
<artifactId>slf4j-api</artifactId>
110-
<version>1.7.5</version>
111-
</dependency>
112-
<dependency>
113-
<groupId>org.slf4j</groupId>
114-
<artifactId>slf4j-log4j12</artifactId>
115-
<version>1.7.5</version>
116-
</dependency>
117-
</dependencies>
118-
119-
<build>
120-
<plugins>
121-
<plugin>
122-
<artifactId>maven-assembly-plugin</artifactId>
123-
<configuration>
124-
<descriptorRefs>
125-
<descriptorRef>jar-with-dependencies</descriptorRef>
126-
</descriptorRefs>
127-
<finalName>cosmosdb-cassandra-examples</finalName>
128-
<appendAssemblyId>false</appendAssemblyId>
129-
</configuration>
130-
<executions>
131-
<execution>
132-
<id>make-assembly</id>
133-
<phase>package</phase>
134-
<goals>
135-
<goal>single</goal>
136-
</goals>
137-
</execution>
138-
</executions>
139-
</plugin>
140-
<plugin>
141-
<groupId>org.apache.maven.plugins</groupId>
142-
<artifactId>maven-compiler-plugin</artifactId>
143-
<configuration>
144-
<source>1.8</source>
145-
<target>1.8</target>
146-
</configuration>
147-
</plugin>
148-
</plugins>
149-
</build>
150-
56+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
57+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
58+
<modelVersion>4.0.0</modelVersion>
59+
60+
<groupId>com.azure.cosmosdb.cassandra</groupId>
61+
<artifactId>cosmosdb-cassandra-examples</artifactId>
62+
<version>1.0-SNAPSHOT</version>
63+
<dependencies>
64+
<dependency>
65+
<groupId>org.apache.flink</groupId>
66+
<artifactId>flink-java</artifactId>
67+
<version>1.17.0</version>
68+
</dependency>
69+
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java -->
70+
<dependency>
71+
<groupId>org.apache.flink</groupId>
72+
<artifactId>flink-streaming-java</artifactId>
73+
<version>1.17.0</version>
74+
</dependency>
75+
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
76+
<dependency>
77+
<groupId>org.apache.flink</groupId>
78+
<artifactId>flink-clients</artifactId>
79+
<version>1.17.0</version>
80+
</dependency>
81+
<dependency>
82+
<groupId>org.apache.flink</groupId>
83+
<artifactId>flink-connector-files</artifactId>
84+
<version>1.17.0</version>
85+
</dependency>
86+
<dependency>
87+
<groupId>org.apache.flink</groupId>
88+
<artifactId>flink-connector-kafka</artifactId>
89+
<version>1.17.0</version>
90+
</dependency>
91+
<dependency>
92+
<groupId>com.datastax.cassandra</groupId>
93+
<artifactId>cassandra-driver-core</artifactId>
94+
<version>3.3.0</version>
95+
</dependency>
96+
<dependency>
97+
<groupId>com.datastax.cassandra</groupId>
98+
<artifactId>cassandra-driver-mapping</artifactId>
99+
<version>3.1.4</version>
100+
</dependency>
101+
<dependency>
102+
<groupId>com.datastax.cassandra</groupId>
103+
<artifactId>cassandra-driver-extras</artifactId>
104+
<version>3.1.4</version>
105+
</dependency>
106+
<dependency>
107+
<groupId>org.slf4j</groupId>
108+
<artifactId>slf4j-api</artifactId>
109+
<version>1.7.5</version>
110+
</dependency>
111+
<dependency>
112+
<groupId>org.slf4j</groupId>
113+
<artifactId>slf4j-log4j12</artifactId>
114+
<version>1.7.5</version>
115+
</dependency>
116+
</dependencies>
117+
118+
<build>
119+
<plugins>
120+
<plugin>
121+
<artifactId>maven-assembly-plugin</artifactId>
122+
<configuration>
123+
<descriptorRefs>
124+
<descriptorRef>jar-with-dependencies</descriptorRef>
125+
</descriptorRefs>
126+
<finalName>cosmosdb-cassandra-examples</finalName>
127+
<appendAssemblyId>false</appendAssemblyId>
128+
</configuration>
129+
<executions>
130+
<execution>
131+
<id>make-assembly</id>
132+
<phase>package</phase>
133+
<goals>
134+
<goal>single</goal>
135+
</goals>
136+
</execution>
137+
</executions>
138+
</plugin>
139+
<plugin>
140+
<groupId>org.apache.maven.plugins</groupId>
141+
<artifactId>maven-compiler-plugin</artifactId>
142+
<configuration>
143+
<source>1.8</source>
144+
<target>1.8</target>
145+
</configuration>
146+
</plugin>
147+
</plugins>
148+
</build>
149+
151150
</project>
152-
153151
```
152+
154153
**Cosmos DB for Apache Cassandra's connection configuration**
155154

156155
You're required to update your host-name and user-name, and keys in the below snippet.
@@ -430,7 +429,7 @@ public class CassandraDemo {
430429

431430
### Building the project
432431

433-
Run **mvn clean install** from azure-cosmos-db-cassandra-java-getting-started-main folder to build the project. This command generates cosmosdb-cassandra-examples.jar under target folder
432+
Run **mvn clean install** from azure-cosmos-db-cassandra-java-getting-started-main folder to build the project. This command generates cosmosdb-cassandra-examples.jar under target folder.
434433

435434
```
436435
root@flinkvm:/home/flinkvm/azure-cosmos-db-cassandra-java-getting-started-main/target# ll
@@ -465,7 +464,7 @@ bin/flink run -c com.azure.cosmosdb.cassandra.examples.UserProfile -j cosmosdb-c
465464

466465
## Sink Kafka Topics into Cosmos DB for Apache Cassandra
467466

468-
Run CassandraDemo class to sink Kafka topic into Cosmos DB for Apache Cassandra
467+
Run CassandraDemo class to sink Kafka topic into Cosmos DB for Apache Cassandra.
469468

470469
```
471470
bin/flink run -c com.azure.cosmosdb.cassandra.examples.CassandraDemo -j cosmosdb-cassandra-examples.jar
@@ -475,13 +474,13 @@ bin/flink run -c com.azure.cosmosdb.cassandra.examples.CassandraDemo -j cosmosdb
475474

476475
## Validate Apache Flink Job Submission
477476

478-
Check job on Flink Web UI on HDInsight on AKS Cluster
477+
Check job on Flink Web UI on HDInsight on AKS Cluster.
479478

480479
:::image type="content" source="./media/cosmos-db-for-apache-cassandra/check-output-on-flink-ui.png" alt-text="Screenshot showing how to check the job on HDInsight on AKS Flink UI." lightbox="./media/cosmos-db-for-apache-cassandra/check-output-on-flink-ui.png":::
481480

482481
## Producing Messages in Kafka
483482

484-
Produce message into Kafka topic
483+
Produce message into Kafka topic.
485484

486485
``` python
487486
sshuser@hn0-flinkd:~$ cat user.py
@@ -550,4 +549,4 @@ sshuser@hn0-flinkd:~$ python user.py | /usr/hdp/current/kafka-broker/bin/kafka-c
550549
* [Azure Cosmos DB for Apache Cassandra](../../cosmos-db/cassandra/introduction.md).
551550
* [Create a API for Cassandra account in Azure Cosmos DB](../../cosmos-db/cassandra/create-account-java.md)
552551
* [Azure Samples ](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started)
553-
* Apache, Apache Kafka, Kafka, Apache Flink, Flink, Apache Cassandra, Cassandra and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).
552+
* Apache, Apache Kafka, Kafka, Apache Flink, Flink, Apache Cassandra, Cassandra, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

‎articles/hdinsight-aks/flink/flink-web-ssh-on-portal-to-flink-sql.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
11
---
22
title: How to enter the Apache Flink® CLI client using Secure Shell (SSH) on HDInsight on AKS clusters with Azure portal
3-
description: How to enter Apache Flink® SQL & DStream CLI client using webssh on HDInsight on AKS clusters with Azure portal
3+
description: How to enter Apache Flink® SQL & DStream CLI client using webssh on HDInsight on AKS clusters with Azure portal.
44
ms.service: hdinsight-aks
55
ms.topic: how-to
6-
ms.date: 10/27/2023
6+
ms.date: 02/04/2024
77
---
88

99
# Access Apache Flink® CLI client using Secure Shell (SSH) on HDInsight on AKS clusters with Azure portal
1010

1111
[!INCLUDE [feature-in-preview](../includes/feature-in-preview.md)]
1212

13-
This example guides how to enter the Apache Flink CLI client on HDInsight on AKS clusters using SSH on Azure portal, we cover both SQL and Flink DataStream
13+
This example guides how to enter the Apache Flink CLI client on HDInsight on AKS clusters using SSH on Azure portal, we cover both SQL and Flink DataStream.
1414

1515
## Prerequisites
1616
- You're required to select SSH during [creation](./flink-create-cluster-portal.md) of Flink Cluster

‎articles/hdinsight-aks/flink/media/cosmos-db-for-apache-cassandra/check-output-on-flink-ui.png

6.61 KB
Loading

‎articles/hdinsight-aks/flink/media/cosmos-db-for-apache-cassandra/run-cassandra-demo.png

-66.6 KB
Loading

‎articles/hdinsight-aks/flink/media/flink-web-ssh-on-portal-to-flink-sql/create-pod-and-connect-to-web-ssh.png

65 KB
Loading

0 commit comments

Comments
 (0)