Skip to content

Commit 66a8377

Browse files
author
Sreekanth Iyer (Ushta Te Consultancy Services)
committed
Updated new screenshots
1 parent 7cec920 commit 66a8377

File tree

8 files changed

+24
-35
lines changed

8 files changed

+24
-35
lines changed

articles/hdinsight-aks/flink/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -171,18 +171,19 @@ public class KafkaSinkToGen2 {
171171

172172
1. Upload the jar to ABFS.
173173

174-
:::image type="content" source="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/app-mode.png" alt-text="Screenshot showing Flink app mode screen.":::
174+
:::image type="content" source="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/app-mode.png" alt-text="Screenshot showing Flink app mode screen." lightbox="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/app-mode.png":::
175+
175176

176177
1. Pass the job jar information in `AppMode` cluster creation.
177178

178-
:::image type="content" source="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/create-app-mode.png" alt-text="Screenshot showing create app mode.":::
179+
:::image type="content" source="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/create-app-mode.png" alt-text="Screenshot showing create app mode." lightbox="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/-create-app-mode.png":::
179180

180181
> [!NOTE]
181182
> Make sure to add classloader.resolve-order as ‘parent-first’ and hadoop.classpath.enable as `true`
182183
183184
1. You can see the job running.
184185

185-
:::image type="content" source="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/flink-ui.png" alt-text="Screenshot showing Flink UI.":::
186+
:::image type="content" source="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/flink-ui.png" alt-text="Screenshot showing Flink UI." lightbox="./media/assign-kafka-topic-event-message-to-azure-data-lake-storage-gen2/flink-ui.png":::
186187

187188

188189
**Validate streaming data on ADLS Gen2**

articles/hdinsight-aks/flink/flink-configuration-management.md

Lines changed: 20 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,28 @@
11
---
22
title: Apache Flink® Configuration Management in HDInsight on AKS
3-
description: Learn about Apache Flink Configuration Management in HDInsight on AKS
3+
description: Learn about Apache Flink Configuration Management in HDInsight on AKS.
44
ms.service: hdinsight-aks
55
ms.topic: tutorial
6-
ms.date: 08/29/2023
6+
ms.date: 04/25/2024
77
---
88

99
# Apache Flink® Configuration management in HDInsight on AKS
1010

1111
[!INCLUDE [feature-in-preview](../includes/feature-in-preview.md)]
1212

13-
HDInsight on AKS provides a set of default configurations of Apache Flink for most properties and a few based on common application profiles. However, in case you're required to tweak Flink configuration properties to improve performance for certain applications with state usage, parallelism, or memory settings, you can change certain properties at cluster level using **Configuration management** section in HDInsight on AKS cluster.
13+
HDInsight on AKS provides a set of default configurations of Apache Flink for most properties and a few based on common application profiles. However, in case you're required to tweak Flink configuration properties to improve performance for certain applications with state usage, parallelism, or memory settings, you can change Flink job configuration using Flink Jobs Section in HDInsight on AKS cluster.
1414

15-
1. Go to **Configuration Management** section on your Apache Flink cluster page
15+
1. Go To Settings > Flink Jobs > Click on Update.
1616

17-
:::image type="content" source="./media/flink-configuration-management/configuration-page-revised.png" alt-text="Screenshot showing Apache Flink Configuration Management page." lightbox="./media/flink-configuration-management/configuration-page-revised.png":::
17+
:::image type="content" source="./media/update-page.png" alt-text="Screenshot showing update page." lightbox="./media/flink-configuration-management/update-page.png":::
1818

19-
2. Update **configurations** as required at *Cluster level*
19+
1. Click on **+ Add a row** to edit configuration.
2020

21-
:::image type="content" source="./media/flink-configuration-management/update-configuration-revised.png" alt-text="Screenshot showing Apache Flink Update configuration page." lightbox="./media/flink-configuration-management/update-configuration-revised.png":::
21+
:::image type="content" source="./media/flink-configuration-management/update-job.png" alt-text="Screenshot update job." lightbox="./media/flink-configuration-management/update-job.png":::
2222

2323
Here the checkpoint interval is changed at *Cluster level*.
2424

25-
3. Update the changes by clicking **OK** and then **Save**.
25+
1. Update the changes by clicking **OK** and then **Save**.
2626

2727
Once saved, the new configurations get updated in a few minutes (~5 minutes).
2828

@@ -32,7 +32,7 @@ Configurations, which can be updated using Configuration Management Settings
3232

3333
The default settings for the process memory size of or job manager and task manager would be the memory configured by the user during cluster creation.
3434

35-
This size can be configured by using the below configuration property. In-order to change task manager process memory, use this configuration
35+
This size can be configured by using the below configuration property. In-order to change task manager process memory, use this configuration.
3636

3737
`taskmanager.memory.process.size : <value>`
3838

@@ -48,28 +48,28 @@ For job manager,
4848
4949
## Checkpoint Interval
5050

51-
The checkpoint interval determines how often Flink triggers a checkpoint. it's defined in milliseconds and can be set using the following configuration property:
51+
The checkpoint interval determines how often Flink triggers a checkpoint. Defined in milliseconds and can be set using the following configuration property
5252

5353
`execution.checkpoint.interval: <value>`
5454

5555
Default setting is 60,000 milliseconds (1 min), this value can be changed as desired.
5656

5757
## State Backend
5858

59-
The state backend determines how Flink manages and persists the state of your application. It impacts how checkpoints are stored. You can configure the `state backend using the following property:
59+
The state backend determines how Flink manages and persists the state of your application. It impacts how checkpoints stored. You can configure the `state backend using the following property:
6060

6161
`state.backend: <value>`
6262

63-
By default Apache Flink clusters in HDInsight on AKS use Rocks db
63+
By default Apache Flink clusters in HDInsight on AKS use Rocks DB.
6464

6565
## Checkpoint Storage Path
6666

6767
We allow persistent checkpoints by default by storing the checkpoints in `abfs` storage as configured by the user. Even if the job fails, since the checkpoints are persisted, it can be easily started with the latest checkpoint.
6868

6969
`state.checkpoints.dir: <path>`
70-
Replace `<path>` with the desired path where the checkpoints are stored.
70+
Replace `<path>` with the desired path where the checkpoints stored.
7171

72-
By default, it's stored in the storage account (ABFS), configured by the user. This value can be changed to any path desired as long as the Flink pods can access it.
72+
By default, stored in the storage account (ABFS), configured by the user. This value can be changed to any path desired as long as the Flink pods can access it.
7373

7474
## Maximum Concurrent Checkpoints
7575

@@ -88,8 +88,8 @@ Replace `<value>` with desired maximum number. By default we retain maximum five
8888

8989
We allow persistent savepoints by default by storing the savepoints in `abfs` storage (as configured by the user). If the user wants to stop and later start the job with a particular savepoint, they can configure this location.
9090
state.checkpoints.dir: `<path>`
91-
Replace` <path>` with the desired path where the savepoints are stored.
92-
By default, it's stored in the storage account, configured by the user. (We support ABFS). This value can be changed to any path desired as long as the Flink pods can access it.
91+
Replace` <path>` with the desired path where the savepoints stored.
92+
By default, stored in the storage account, configured by the user. (We support ABFS). This value can be changed to any path desired as long as the Flink pods can access it.
9393

9494
## Job manager high availability
9595

@@ -103,25 +103,13 @@ Even if the jobs fail abruptly, if the checkpoints are happening continuously, t
103103
Change the job strategy in between?
104104
There are use cases, where the job needs to be modified while in production due to some job level bug. During that time, the user can stop the job, which would automatically take a savepoint and save it in savepoint location.
105105

106-
`bin/flink stop <JOBID>`
106+
1. Click on `savepoint` and wait for `savepoint` to be completed.
107107

108-
Example:
109-
110-
```
111-
root [ ~ ]# ./bin/flink stop 60bdf21d9bc3bc65d63bc3d8fc6d5c54
112-
Suspending job "60bdf21d9bc3bc65d63bc3d8fc6d5c54" with a CANONICAL savepoint.
113-
Savepoint completed. Path: abfs://[email protected]/8255a11812144c28b4ddf1068460c96b/savepoints/savepoint-60bdf2-7717485d15e3
114-
```
115-
116-
Later the user can start the job with bug fix pointing to the savepoint.
108+
:::image type="content" source="./media/flink-configuration-management/save-point.png" alt-text="Screenshot showing save point options." lightbox="./media/flink-configuration-management/save-point.png":::
117109

118-
```
119-
./bin/flink run <JOB_JAR> -d <SAVEPOINT_LOC>
120-
root [ ~ ]# ./bin/flink run examples/streaming/StateMachineExample.jar -s abfs://[email protected]/8255a11812144c28b4ddf1068460c96b/savepoints/savepoint-60bdf2-7717485d15e3
121-
```
122-
Usage with built-in data generator: StateMachineExample [--error-rate `<probability-of-invalid-transition>] [--sleep <sleep-per-record-in-ms>]`
110+
1. After savepoint completion, click on start and Start Job Tab will appear. Select the savepoint name from the dropdown. Edit any configurations if necessary. And click **OK**.
123111

124-
Usage with Kafka: `StateMachineExample --kafka-topic <topic> [--brokers <brokers>]`
112+
:::image type="content" source="./media/flink-configuration-management/start-job.png" alt-text="Screenshot showing how to start job." lightbox="./media/flink-configuration-management/start.job.png":::
125113

126114
Since savepoint is provided in the job, the Flink knows from where to start processing the data.
127115

68.8 KB
Loading
103 KB
Loading
136 KB
Loading
74.2 KB
Loading

0 commit comments

Comments
 (0)