Skip to content

Commit 3978072

Browse files
[fix]: update titles and improve clarity in Kafka deployment documentation
1 parent 239951a commit 3978072

File tree

5 files changed

+88
-74
lines changed

5 files changed

+88
-74
lines changed

content/learning-paths/servers-and-cloud-computing/kafka-azure/_index.md

Lines changed: 8 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,19 @@
11
---
2-
title: Deploy Kafka on the Microsoft Azure Cobalt 100 processors
3-
4-
draft: true
5-
cascade:
6-
draft: true
2+
title: Deploy Apache Kafka on Arm-based Microsoft Azure Cobalt 100 virtual machines
73

84
minutes_to_complete: 30
95

10-
who_is_this_for: This is an advanced topic designed for software developers looking to migrate their Kafka workloads from x86_64 to Arm-based platforms, specifically on the Microsoft Azure Cobalt 100 processors.
6+
who_is_this_for: This is an advanced topic for developers looking to migrate their Apache Kafka workloads from x86_64 to Arm-based platforms, specifically on Microsoft Azure Cobalt 100 (arm64) virtual machines.
117

128
learning_objectives:
13-
- Provision an Azure Arm64 virtual machine using Azure console, with Ubuntu Pro 24.04 LTS as the base image.
14-
- Deploy Kafka on the Ubuntu virtual machine.
15-
- Perform Kafka baseline testing and benchmarking on Arm64 virtual machines.
9+
- Provision an Azure Arm64 virtual machine using Azure console, with Ubuntu Pro 24.04 LTS as the base image
10+
- Deploy Kafka on an Ubuntu virtual machine
11+
- Perform Kafka baseline testing and benchmarking on Arm64 virtual machines
1612

1713
prerequisites:
18-
- A [Microsoft Azure](https://azure.microsoft.com/) account with access to Cobalt 100 based instances (Dpsv6).
19-
- Basic understanding of Linux command line.
20-
- Familiarity with the [Apache Kafka architecture](https://kafka.apache.org/) and deployment practices on Arm64 platforms.
14+
- A [Microsoft Azure](https://azure.microsoft.com/) account with access to Cobalt 100 based instances (Dpsv6)
15+
- Basic understanding of Linux command line
16+
- Familiarity with the [Apache Kafka architecture](https://kafka.apache.org/) and deployment practices on Arm64 platforms
2117

2218
author: Pareena Verma
2319

content/learning-paths/servers-and-cloud-computing/kafka-azure/background.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ layout: "learningpathall"
88

99
## Cobalt 100 Arm-based processor
1010

11-
Azure’s Cobalt 100 is built on Microsoft's first-generation, in-house Arm-based processor: the Cobalt 100. Designed entirely by Microsoft and based on Arm’s Neoverse N2 architecture, this 64-bit CPU delivers improved performance and energy efficiency across a broad spectrum of cloud-native, scale-out Linux workloads. These include web and application servers, data analytics, open-source databases, caching systems, and more. Running at 3.4 GHz, the Cobalt 100 processor allocates a dedicated physical core for each vCPU, ensuring consistent and predictable performance.
11+
Azure’s Cobalt 100 is built on Microsoft's first-generation, in-house Arm-based processor: the Cobalt 100. Designed entirely by Microsoft and based on Arm’s Neoverse N2 architecture, this 64-bit CPU delivers improved performance and energy efficiency across a broad spectrum of cloud-native, scale-out Linux workloads. These include web and application servers, data analytics, open-source databases, caching systems, and more. Running at 3.4 GHz, the Cobalt 100 processor allocates a dedicated physical core for each virtual CPU (vCPU), ensuring consistent and predictable performance.
1212

1313
To learn more about Cobalt 100, refer to the blog [Announcing the preview of new Azure virtual machine based on the Azure Cobalt 100 processor](https://techcommunity.microsoft.com/blog/azurecompute/announcing-the-preview-of-new-azure-vms-based-on-the-azure-cobalt-100-processor/4146353).
1414

@@ -17,4 +17,4 @@ Apache Kafka is a high-performance, open-source distributed event streaming plat
1717

1818
It allows you to publish, subscribe to, store, and process streams of records in a fault-tolerant and scalable manner. Kafka stores data in topics, which are partitioned and replicated across a cluster to ensure durability and high availability.
1919

20-
Kafka is widely used for messaging, log aggregation, event sourcing, real-time analytics, and integrating large-scale data systems. Learn more from the [Apache Kafka official website](https://kafka.apache.org/) and its [official documentation](https://kafka.apache.org/documentation).
20+
Kafka is widely used for messaging, log aggregation, event sourcing, real-time analytics, and integrating large-scale data systems. Learn more from the [Apache Kafka official website](https://kafka.apache.org/) and the [Apache Kafka documentation](https://kafka.apache.org/documentation).

content/learning-paths/servers-and-cloud-computing/kafka-azure/baseline.md

Lines changed: 64 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Baseline Testing
2+
title: Run baseline testing with Kafka on Azure Arm VM
33
weight: 5
44

55
### FIXED, DO NOT MODIFY
@@ -12,60 +12,64 @@ After installing Apache Kafka 4.1.0 on your Azure Cobalt 100 Arm64 virtual machi
1212
Kafka 4.1.0 introduces KRaft mode (Kafka Raft Metadata mode), which integrates the control and data planes, eliminating the need for ZooKeeper.
1313
This simplifies deployment, reduces latency, and provides a unified, self-managed Kafka cluster architecture.
1414

15-
To perform this baseline test, you will use four terminal sessions:
16-
Terminal 1: Start the Kafka broker (in KRaft mode).
17-
Terminal 2: Create a topic.
18-
Terminal 3: Send messages (Producer).
19-
Terminal 4: Read messages (Consumer).
15+
To run this baseline test, open four terminal sessions:
2016

21-
### Initial Setup: Configure & Format KRaft
22-
KRaft (Kafka Raft) replaces ZooKeeper by embedding metadata management directly into the Kafka broker.
23-
This improves scalability, reduces external dependencies, and speeds up controller failover in distributed clusters.
24-
Before starting Kafka in KRaft mode, configure and initialize the storage directory. These steps are required only once per broker.
17+
- **Terminal 1:** Start the Kafka broker in KRaft mode.
18+
- **Terminal 2:** Create a topic.
19+
- **Terminal 3:** Send messages as the producer.
20+
- **Terminal 4:** Read messages as the consumer.
2521

26-
1. Edit the Configuration File
27-
Open the Kafka configuration file in an editor:
22+
Each terminal has a specific role, helping you verify that Kafka works end-to-end on your Arm64 VM.
23+
## Configure and format KRaft
2824

29-
```console
30-
vi /opt/kafka/config/server.properties
31-
```
25+
KRaft (Kafka Raft) mode replaces ZooKeeper by managing metadata directly within the Kafka broker. This change improves scalability, reduces external dependencies, and speeds up controller failover in distributed clusters.
3226

33-
2. Add or Modify KRaft Properties
34-
Ensure the following configuration entries are present for a single-node KRaft setup:
27+
Before you start Kafka in KRaft mode, you need to configure the broker and initialize the storage directory. You only need to do this once for each broker.
3528

36-
```java
37-
process.roles=controller,broker
38-
node.id=1
39-
controller.quorum.voters=1@localhost:9093
40-
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
41-
advertised.listeners=PLAINTEXT://localhost:9092
42-
log.dirs=/tmp/kraft-combined-logs
43-
```
44-
This configuration file sets up a single Kafka server to act as both a controller (managing cluster metadata) and a broker (handling data), running in KRaft mode. It defines the node's unique ID and specifies the local host as the sole participant in the controller quorum.
4529

46-
3. Format the Storage Directory
47-
Format the metadata storage directory using the kafka-storage.sh tool. This initializes KRaft’s internal Raft logs with a unique cluster ID.
30+
- **Edit the configuration file**
31+
Open the Kafka configuration file in an editor:
4832

49-
```console
50-
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties
51-
```
52-
You should see output similar to:
33+
```console
34+
vi /opt/kafka/config/server.properties
35+
```
5336

54-
```output
55-
Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1.
56-
```
57-
This confirms that the Kafka storage directory has been successfully formatted and that the broker is ready to start in KRaft mode.
37+
- **Add or modify KRaft properties**
38+
Ensure the following configuration entries are present for a single-node KRaft setup:
39+
40+
```java
41+
process.roles=controller,broker
42+
node.id=1
43+
controller.quorum.voters=1@localhost:9093
44+
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
45+
advertised.listeners=PLAINTEXT://localhost:9092
46+
log.dirs=/tmp/kraft-combined-logs
47+
```
48+
This configuration file sets up a single Kafka server to act as both a controller (managing cluster metadata) and a broker (handling data), running in KRaft mode. It defines the node's unique ID and specifies the local host as the sole participant in the controller quorum.
49+
50+
- **Format the storage directory**
51+
Format the metadata storage directory using the kafka-storage.sh tool. This initializes KRaft’s internal Raft logs with a unique cluster ID.
52+
53+
```console
54+
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties
55+
```
56+
You should see output similar to:
57+
58+
```output
59+
Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1.
60+
```
61+
This confirms that the Kafka storage directory has been successfully formatted and that the broker is ready to start in KRaft mode.
5862

5963
## Perform the Baseline Test
6064
With Kafka 4.1.0 installed and configured in KRaft mode, you’re now ready to run a baseline test to verify that the Kafka broker starts correctly, topics can be created, and message flow works as expected.
6165

6266
You’ll use multiple terminals for this test:
63-
Terminal 1: Start the Kafka broker.
64-
Terminal 2: Create and verify a topic.
65-
Terminal 3: Send messages (Producer).
66-
Terminal 4: Read messages (Consumer).
67+
Terminal 1: start the Kafka broker
68+
Terminal 2: create and verify a topic
69+
Terminal 3: send messages (Producer)
70+
Terminal 4: read messages (Consumer)
6771

68-
### Terminal 1 – Start Kafka Broker
72+
## Terminal 1 - start Kafka broker
6973
Start the Kafka broker (the main server process responsible for managing topics and handling messages) in KRaft mode:
7074

7175
```console
@@ -74,7 +78,7 @@ bin/kafka-server-start.sh config/server.properties
7478
```
7579
Keep this terminal open and running. The broker process must stay active for all subsequent commands.
7680

77-
### Terminal 2 – Create a Topic
81+
## Terminal 2 - create a Topic
7882
Open a new terminal window. Create a topic named test-topic-kafka, which acts as a logical channel where producers send and consumers receive messages:
7983

8084
```console
@@ -87,8 +91,21 @@ You should see output similar to:
8791
Created topic test-topic-kafka.
8892
```
8993

90-
**Verify Topic Creation**
91-
List available topics to confirm that your new topic was created successfully:
94+
## Verify topic creation
95+
List available topics to confirm that your new topic was created successfully. Run the following command:
96+
97+
```console
98+
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
99+
```
100+
101+
The expected output is:
102+
103+
```output
104+
__consumer_offsets
105+
test-topic-kafka
106+
```
107+
108+
If you see `test-topic-kafka` in the list, your topic was created and is ready for use.
92109

93110
```console
94111
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
@@ -102,7 +119,7 @@ test-topic-kafka
102119
Kafka is now running, and you’ve successfully created and verified a topic.
103120
Next, you’ll use Terminal 3 to produce messages and Terminal 4 to consume messages, completing the baseline functional test on your Arm64 environment.
104121

105-
### Terminal 3 – Console Producer (Write Message)
122+
## Terminal 3 - console producer (write message)
106123
In this step, you’ll start the Kafka Producer, which publishes messages to the topic test-topic-kafka. The producer acts as the data source, sending messages to the Kafka broker.
107124

108125
```console
@@ -117,8 +134,8 @@ hello from azure arm vm
117134
```
118135
Each line you type is sent as a message to the Kafka topic and stored on disk by the broker.
119136

120-
### Terminal 4 – Console Consumer (Read Message)
121-
Next, open another terminal and start the Kafka Consumer, which subscribes to the same topic (test-topic-kafka) and reads messages from the beginning of the log.
137+
## Terminal 4 - console consumer (read message)
138+
Next, open another terminal and start the Kafka Consumer, which subscribes to the same topic (test-topic-kafka) and reads messages from the beginning of the log:
122139

123140
```console
124141
cd /opt/kafka

content/learning-paths/servers-and-cloud-computing/kafka-azure/benchmarking.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Benchmarking with Official Kafka Tools
2+
title: Benchmark with official Kafka tools
33
weight: 6
44

55
### FIXED, DO NOT MODIFY
@@ -13,7 +13,7 @@ Apache Kafka includes official performance testing utilities that allow you to m
1313
## Steps for Kafka Benchmarking
1414

1515
Before running the benchmarks, make sure your Kafka broker is already active in a separate terminal (as configured in the previous section).
16-
Now open two new terminal sessionsone for running the producer benchmark and another for the consumer benchmark.
16+
Now open two new terminal sessions; one for running the producer benchmark, and the other for the consumer benchmark.
1717

1818
### Terminal A - Producer Benchmark
1919

content/learning-paths/servers-and-cloud-computing/kafka-azure/create-instance.md

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Create an Arm based cloud virtual machine using Microsoft Cobalt 100 CPU
2+
title: Create an Arm-based cloud virtual machine using Microsoft Cobalt 100 CPU
33
weight: 3
44

55
### FIXED, DO NOT MODIFY
@@ -8,21 +8,22 @@ layout: learningpathall
88

99
## Introduction
1010

11-
There are several ways to create an Arm-based Cobalt 100 virtual machine : the Microsoft Azure console, the Azure CLI tool, or using your choice of IaC (Infrastructure as Code). This guide will use the Azure console to create a virtual machine with Arm-based Cobalt 100 Processor.
11+
You can create an Arm-based Cobalt 100 virtual machine in several ways: using the Microsoft Azure portal, the Azure CLI, or Infrastructure as Code (IaC) tools. This Learning Path uses the Azure portal to walk you through creating a virtual machine with an Arm-based Cobalt 100 processor, which is from the D-Series v6 general-purpose virtual machines.
1212

13-
This learning path focuses on the general-purpose virtual machine of the D series. Please read the guide on [Dpsv6 size series](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dpsv6-series) offered by Microsoft Azure.
13+
These VMs are designed for a wide range of workloads and offer Arm-based performance with the Cobalt 100 CPU. To learn more about the Dpsv6 size series, see the official [Dpsv6 size series guide](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dpsv6-series) from Microsoft Azure.
1414

15-
If you have never used the Microsoft Cloud Platform before, please review the microsoft [guide to Create a Linux virtual machine in the Azure portal](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal?tabs=ubuntu).
15+
If you have never used the Microsoft Cloud Platform before, see the Microsoft guide on how to [Create a Linux virtual machine in the Azure portal](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal?tabs=ubuntu).
1616

17-
#### Create an Arm-based Azure Virtual Machine
17+
## Create an Arm-based Azure Virtual Machine
1818

19-
Creating a virtual machine based on Azure Cobalt 100 is no different from creating any other virtual machine in Azure. To create an Azure virtual machine, launch the Azure portal and navigate to "Virtual Machines".
20-
1. Select "Create", and click on "Virtual Machine" from the drop-down list.
21-
2. Inside the "Basic" tab, fill in the Instance details such as "Virtual machine name" and "Region".
22-
3. Choose the image for your virtual machine (for example, Ubuntu Pro 24.04 LTS) and select “Arm64” as the VM architecture.
23-
4. In the “Size” field, click on “See all sizes” and select the D-Series v6 family of virtual machines. Select “D4ps_v6” from the list.
19+
Creating a virtual machine based on Azure Cobalt 100 is no different from creating any other virtual machine in Azure. To create an Azure virtual machine, launch the Azure portal and navigate to **Virtual Machines**.
2420

25-
![Azure portal VM creation — Azure Cobalt 100 Arm64 virtual machine (D4ps_v6) alt-text#center](images/instance.png "Figure 1: Select the D-Series v6 family of virtual machines")
21+
- Select **Create**, and click on **Virtual Machine** from the drop-down list.
22+
- Inside the **Basic** tab, fill in the Instance details such as **Virtual machine name** and **Region**.
23+
- Choose the image for your virtual machine (for example, Ubuntu Pro 24.04 LTS) and select **Arm64** as the VM architecture.
24+
- In the **Size** field, click on **See all sizes** and select the D-Series v6 family of virtual machines. Select **D4ps_v6** from the list.
25+
26+
![Azure portal showing the selection of the D-Series v6 family of virtual machines, with D4ps_v6 highlighted as the chosen size. The interface displays a list of available VM sizes, including CPU, memory, and pricing details. The wider environment is the Azure portal's virtual machine creation workflow, with a clean and organized layout. The tone is neutral and informative, focused on guiding users through the selection process. Visible text includes D-Series v6, D4ps_v6, CPU, memory, and price columns.](images/instance.png "Selecting the D-Series v6 family of virtual machines")
2627

2728
5. Select "SSH public key" as an Authentication type. Azure will automatically generate an SSH key pair for you and allow you to store it for future use. It is a fast, simple, and secure way to connect to your virtual machine.
2829
6. Fill in the Administrator username for your VM.

0 commit comments

Comments
 (0)