Skip to content

Commit 7d75d41

Browse files
pareenavermapareenaverma
authored
Merge pull request #2272 from odidev/kafka
Deploy Kafka on the Microsoft Azure Cobalt 100 processors
2 parents 3ada49b + ceef776 commit 7d75d41

File tree

12 files changed

+410
-0
lines changed

12 files changed

+410
-0
lines changed

‎content/learning-paths/servers-and-cloud-computing/kafka-azure/_index.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: Deploy Kafka on the Microsoft Azure Cobalt 100 processors
3+
4+
draft: true
5+
cascade:
6+
draft: true
7+
8+
minutes_to_complete: 30
9+
10+
who_is_this_for: This Learning Path is designed for software developers looking to migrate their Kafka workloads from x86_64 to Arm-based platforms, specifically on the Microsoft Azure Cobalt 100 processors.
11+
12+
learning_objectives:
13+
- Provision an Azure Arm64 virtual machine using Azure console, with Ubuntu Pro 24.04 LTS as the base image.
14+
- Deploy Kafka on the Ubuntu virtual machine.
15+
- Perform Kafka baseline testing and benchmarking on both x86_64 and Arm64 virtual machines.
16+
17+
prerequisites:
18+
- A [Microsoft Azure](https://azure.microsoft.com/) account with access to Cobalt 100 based instances (Dpsv6).
19+
- Basic understanding of Linux command line.
20+
- Familiarity with the [Apache Kafka architecture](https://kafka.apache.org/) and deployment practices on Arm64 platforms.
21+
22+
author: Jason Andrews
23+
24+
### Tags
25+
skilllevels: Advanced
26+
subjects: Storage
27+
cloud_service_providers: Microsoft Azure
28+
29+
armips:
30+
- Neoverse
31+
32+
tools_software_languages:
33+
- Kafka
34+
- kafka-producer-perf-test.sh
35+
- kafka-consumer-perf-test.sh
36+
37+
operatingsystems:
38+
- Linux
39+
40+
further_reading:
41+
- resource:
42+
title: Kafka Manual
43+
link: https://kafka.apache.org/documentation/
44+
type: documentation
45+
- resource:
46+
title: Kafka Performance Tool
47+
link: https://codemia.io/knowledge-hub/path/use_kafka-producer-perf-testsh_how_to_set_producer_config_at_kafka_210-0820
48+
type: documentation
49+
- resource:
50+
title: Kafka on Azure
51+
link: https://learn.microsoft.com/en-us/samples/azure/azure-quickstart-templates/kafka-ubuntu-multidisks/
52+
type: documentation
53+
54+
55+
### FIXED, DO NOT MODIFY
56+
# ================================================================================
57+
weight: 1 # _index.md always has weight of 1 to order correctly
58+
layout: "learningpathall" # All files under learning paths have this same wrapper
59+
learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
60+
---

‎content/learning-paths/servers-and-cloud-computing/kafka-azure/_next-steps.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
# ================================================================================
3+
# FIXED, DO NOT MODIFY THIS FILE
4+
# ================================================================================
5+
weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation.
6+
title: "Next Steps" # Always the same, html page title.
7+
layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing.
8+
---

‎content/learning-paths/servers-and-cloud-computing/kafka-azure/background.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
title: "Overview"
3+
4+
weight: 2
5+
6+
layout: "learningpathall"
7+
---
8+
9+
## Cobalt 100 Arm-based processor
10+
11+
Azure’s Cobalt 100 is built on Microsoft's first-generation, in-house Arm-based processor: the Cobalt 100. Designed entirely by Microsoft and based on Arm’s Neoverse N2 architecture, this 64-bit CPU delivers improved performance and energy efficiency across a broad spectrum of cloud-native, scale-out Linux workloads. These include web and application servers, data analytics, open-source databases, caching systems, and more. Running at 3.4 GHz, the Cobalt 100 processor allocates a dedicated physical core for each vCPU, ensuring consistent and predictable performance.
12+
13+
To learn more about Cobalt 100, refer to the blog [Announcing the preview of new Azure virtual machine based on the Azure Cobalt 100 processor](https://techcommunity.microsoft.com/blog/azurecompute/announcing-the-preview-of-new-azure-vms-based-on-the-azure-cobalt-100-processor/4146353).
14+
15+
## Apache Kafka
16+
Apache Kafka is a high-performance, open-source distributed event streaming platform designed for building real-time data pipelines and streaming applications.
17+
18+
It allows you to publish, subscribe to, store, and process streams of records in a fault-tolerant and scalable manner. Kafka stores data in topics, which are partitioned and replicated across a cluster to ensure durability and high availability.
19+
20+
Kafka is widely used for messaging, log aggregation, event sourcing, real-time analytics, and integrating large-scale data systems. Learn more from the [Apache Kafka official website](https://kafka.apache.org/) and its [official documentation](https://kafka.apache.org/documentation).

‎content/learning-paths/servers-and-cloud-computing/kafka-azure/baseline.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
---
2+
title: Baseline Testing
3+
weight: 5
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Run a Baseline test with Kafka
10+
11+
After installing Kafka on your Arm64 virtual machine, you can perform a simple baseline test to validate that Kafka runs correctly and produces the expected output.
12+
13+
Kafka 4.1.0 uses **KRaft**, which integrates the control and data planes, eliminating the need for a separate ZooKeeper instance.
14+
15+
We need 4 terminals to complete this test. The first will start the Kafka server, the second will create a topic, and the final two will send and receive messages, respectively.
16+
17+
### Initial Setup: Configure & Format KRaft
18+
**KRaft** is Kafka's new metadata protocol that integrates the responsibilities of ZooKeeper directly into Kafka, simplifying deployment and improving scalability by making the brokers self-managing.
19+
20+
First, you must configure your `server.properties` file for KRaft and format the storage directory. These steps are done only once.
21+
22+
**1. Edit the Configuration File**: Open your `server.properties` file.
23+
24+
```console
25+
nano /opt/kafka/config/server.properties
26+
```
27+
28+
**2. Add/Modify KRaft Properties:** Ensure the following lines are present and correctly configured for a single-node setup.
29+
30+
This configuration file sets up a single Kafka server to act as both a **controller** (managing cluster metadata) and a broker (handling data), running in **KRaft** mode. It defines the node's unique ID and specifies the local host as the sole participant in the **controller** quorum.
31+
32+
```java
33+
process.roles=controller,broker
34+
node.id=1
35+
controller.quorum.voters=1@localhost:9093
36+
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
37+
advertised.listeners=PLAINTEXT://localhost:9092
38+
log.dirs=/tmp/kraft-combined-logs
39+
```
40+
**3. Format the Storage Directory:** Use the `kafka-storage.sh` tool to format the metadata directory.
41+
42+
```console
43+
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties
44+
```
45+
You should see an output similar to:
46+
47+
```output
48+
Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1.
49+
```
50+
51+
Now, Perform the Baseline Test
52+
53+
### Terminal 1 – Start Kafka Broker
54+
This command starts the Kafka broker (the main server that sends and receives messages) in KRaft mode. Keep this terminal open.
55+
56+
```console
57+
cd /opt/kafka
58+
bin/kafka-server-start.sh config/server.properties
59+
```
60+
### Terminal 2 – Create a Topic
61+
This command creates a new Kafka topic named `test-topic-kafka` (like a channel where messages will be stored and shared) with 1 partition and 1 copy (replica).
62+
63+
```console
64+
cd /opt/kafka
65+
bin/kafka-topics.sh --create --topic test-topic-kafka --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
66+
```
67+
You should see output similar to:
68+
69+
```output
70+
Created topic test-topic-kafka.
71+
```
72+
73+
- **Verify topic**
74+
75+
```console
76+
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
77+
```
78+
You should see output similar to:
79+
80+
```output
81+
__consumer_offsets
82+
test-topic-kafka
83+
```
84+
85+
### Terminal 3 – Console Producer (Write Message)
86+
This command starts the **Kafka Producer**, which lets you type and send messages into the `test-topic-kafka` topic. For example, when you type `hello from azure vm`, this message will be delivered to any Kafka consumer subscribed to that topic.
87+
88+
```console
89+
cd /opt/kafka
90+
bin/kafka-console-producer.sh --topic test-topic-kafka --bootstrap-server localhost:9092
91+
```
92+
You should see an empty prompt where you can start typing. Type `hello from azure arm vm` and press **Enter**.
93+
94+
### Terminal 4 – Console Consumer (Read Message)
95+
This command starts the **Kafka Consumer**, which listens to the `test-topic-kafka` topic and displays all messages from the beginning.
96+
97+
```console
98+
cd /opt/kafka
99+
bin/kafka-console-consumer.sh --topic test-topic-kafka --from-beginning --bootstrap-server localhost:9092
100+
```
101+
102+
You should see your message `hello from azure arm vm` displayed in this terminal, confirming that the producer's message was successfully received.
103+
104+
Now you can proceed to benchmarking Kafka’s performance on the Azure Cobalt 100 Arm virtual machine.

‎content/learning-paths/servers-and-cloud-computing/kafka-azure/benchmarking.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
title: Benchmarking with Official Kafka Tools
3+
weight: 6
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Benchmark Kafka on Azure Cobalt 100 Arm-based instances and x86_64 instances
10+
11+
Kafka’s official performance tools (**kafka-producer-perf-test.sh** and **kafka-consumer-perf-test.sh**) let you generate test workloads, measure message throughput, and record end-to-end latency.
12+
13+
## Steps for Kafka Benchmarking
14+
15+
Before starting the benchmark, ensure that the **Kafka broker** are already running in separate terminals.
16+
17+
Now, open two new terminals—one for the **producer benchmark** and another for the **consumer benchmark**.
18+
19+
### Terminal A - Producer Benchmark
20+
21+
The producer benchmark measures how fast Kafka can send messages, reporting throughput and latency percentiles.
22+
23+
```console
24+
cd /opt/kafka
25+
bin/kafka-producer-perf-test.sh \
26+
--topic test-topic-kafka \
27+
--num-records 1000000 \
28+
--record-size 100 \
29+
--throughput -1 \
30+
--producer-props bootstrap.servers=localhost:9092
31+
```
32+
You should see output similar to:
33+
34+
```output
35+
1000000 records sent, 252589.0 records/sec (24.09 MB/sec), 850.85 ms avg latency, 1219.00 ms max latency, 851 ms 50th, 1184 ms 95th, 1210 ms 99th, 1218 ms 99.9th.
36+
```
37+
### Terminal B - Consumer benchmark
38+
39+
The consumer benchmark measures how fast Kafka can read messages from the topic, reporting throughput and total messages consumed.
40+
41+
```console
42+
cd /opt/kafka
43+
bin/kafka-consumer-perf-test.sh \
44+
--topic test-topic-kafka \
45+
--bootstrap-server localhost:9092 \
46+
--messages 1000000 \
47+
--timeout 30000
48+
```
49+
You should see output similar to:
50+
51+
```output
52+
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
53+
2025-09-03 06:07:13:616, 2025-09-03 06:07:17:545, 95.3674, 24.2727, 1000001, 254517.9435, 3354, 575, 165.8564, 1739132.1739
54+
```
55+
56+
## Benchmark Results Table Explained:
57+
58+
- **Messages Processed** – Total number of messages handled during the test.
59+
- **Records/sec** – Rate of messages sent or consumed per second.
60+
- **MB/sec** – Data throughput in megabytes per second.
61+
- **Avg Latency (ms)** – Average delay in sending messages (producer only).
62+
- **Max Latency (ms)** – Longest observed delay in sending messages (producer only).
63+
- **50th (ms)** – Median latency (half the messages were faster, half slower).
64+
- **95th (ms)** – Latency below which 95% of messages were delivered.
65+
- **99th (ms)** – Latency below which 99% of messages were delivered.
66+
- **99.9th (ms)** – Latency below which 99.9% of messages were delivered.
67+
68+
## Benchmark summary on Arm64:
69+
Here is a summary of benchmark results collected on an Arm64 **D4ps_v6 Ubuntu Pro 24.04 LTS virtual machine**.
70+
### Consumer Performance Test
71+
| Metric | Value | Unit |
72+
|-----------------------------|-------------|---------------|
73+
| Total Time Taken | 3.875 | Seconds |
74+
| Data Consumed | 95.3674 | MB |
75+
| Throughput (Data) | 24.6110 | MB/sec |
76+
| Messages Consumed | 1,000,001 | Messages |
77+
| Throughput (Messages) | 258,064.77 | Messages/sec |
78+
| Rebalance Time | 3348 | Milliseconds |
79+
| Fetch Time | 527 | Milliseconds |
80+
| Fetch Throughput (Data) | 180.9629 | MB/sec |
81+
| Fetch Throughput (Messages)| 1,897,535.10| Messages/sec |
82+
83+
### Producer Performance Test
84+
| Metric | Records Sent | Records/sec | Throughput | Average Latency | Maximum Latency | 50th Percentile Latency | 95th Percentile Latency | 99th Percentile Latency | 99.9th Percentile Latency |
85+
|--------|--------------|-------------|------------|-----------------|-----------------|-------------------------|-------------------------|-------------------------|---------------------------|
86+
| Value | 1,000,000 | 257,532.8 | 24.56 | 816.19 | 1237.00 | 799 | 1168 | 1220 | 1231 |
87+
| Unit | Records | Records/sec | MB/sec | ms | ms | ms | ms | ms | ms |
88+
89+
## Benchmark summary on x86_64:
90+
Here is a summary of the benchmark results collected on x86_64 **D4s_v6 Ubuntu Pro 24.04 LTS virtual machine**.
91+
### Consumer Performance Test
92+
| Metric | Value | Unit |
93+
|--------------------|-------------|---------------|
94+
| Total Time Taken | 3.811 | Seconds |
95+
| Data Consumed | 95.3674 | MB |
96+
| Throughput (Data) | 25.0243 | MB/sec |
97+
| Messages Consumed | 1,000,001 | Messages |
98+
| Throughput (Messages) | 262,398.58 | Messages/sec |
99+
| Rebalance Time | 3271 | Milliseconds |
100+
| Fetch Time | 540 | Milliseconds |
101+
| Fetch Throughput (Data) | 176.6064 | MB/sec |
102+
| Fetch Throughput (Messages) | 1,851,853.70| Messages/sec |
103+
104+
### Producer Performance Test
105+
| Metric | Records Sent | Records/sec | Throughput | Average Latency | Maximum Latency | 50th Percentile Latency | 95th Percentile Latency | 99th Percentile Latency | 99.9th Percentile Latency |
106+
|--------|--------------|-------------|------------|-----------------|-----------------|-------------------------|-------------------------|-------------------------|---------------------------|
107+
| Value | 1,000,000 | 242,013.6 | 23.08 | 840.69 | 1351.00 | 832 | 1283 | 1330 | 1350 |
108+
| Unit | Records | Records/sec | MB/sec | ms | ms | ms | ms | ms | ms |
109+
110+
## Benchmark comparison insights
111+
When comparing the results on Arm64 vs x86_64 virtual machines:
112+
113+
114+
- The Kafka **consumer** achieved **25.02 MB/sec throughput**, processing ~**262K messages/sec** with fetch throughput exceeding **1.85M messages/sec**.
115+
- The Kafka **producer** sustained **23.08 MB/sec throughput**, with an average latency of ~**841 ms** and peak latency of ~**1351 ms**.
116+
- These results confirm stable Kafka performance on the **Azure Ubuntu Pro arm64 virtual machine**, validating its suitability for **baseline testing and benchmarking**.
117+
118+
You have now benchmarked Kafka on an Azure Cobalt 100 Arm64 virtual machine and compared results with x86_64.

‎content/learning-paths/servers-and-cloud-computing/kafka-azure/create-instance.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
---
2+
title: Create an Arm based cloud virtual machine using Microsoft Cobalt 100 CPU
3+
weight: 3
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Introduction
10+
11+
There are several ways to create an Arm-based Cobalt 100 virtual machine : the Microsoft Azure console, the Azure CLI tool, or using your choice of IaC (Infrastructure as Code). This guide will use the Azure console to create a virtual machine with Arm-based Cobalt 100 Processor.
12+
13+
This learning path focuses on the general-purpose virtual machine of the D series. Please read the guide on [Dpsv6 size series](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dpsv6-series) offered by Microsoft Azure.
14+
15+
If you have never used the Microsoft Cloud Platform before, please review the microsoft [guide to Create a Linux virtual machine in the Azure portal](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal?tabs=ubuntu).
16+
17+
#### Create an Arm-based Azure Virtual Machine
18+
19+
Creating a virtual machine based on Azure Cobalt 100 is no different from creating any other virtual machine in Azure. To create an Azure virtual machine, launch the Azure portal and navigate to "Virtual Machines".
20+
1. Select "Create", and click on "Virtual Machine" from the drop-down list.
21+
2. Inside the "Basic" tab, fill in the Instance details such as "Virtual machine name" and "Region".
22+
3. Choose the image for your virtual machine (for example, Ubuntu Pro 24.04 LTS) and select “Arm64” as the VM architecture.
23+
4. In the “Size” field, click on “See all sizes” and select the D-Series v6 family of virtual machines. Select “D4ps_v6” from the list.
24+
25+
![Azure portal VM creation — Azure Cobalt 100 Arm64 virtual machine (D4ps_v6) alt-text#center](images/instance.png "Figure 1: Select the D-Series v6 family of virtual machines")
26+
27+
5. Select "SSH public key" as an Authentication type. Azure will automatically generate an SSH key pair for you and allow you to store it for future use. It is a fast, simple, and secure way to connect to your virtual machine.
28+
6. Fill in the Administrator username for your VM.
29+
7. Select "Generate new key pair", and select "RSA SSH Format" as the SSH Key Type. RSA could offer better security with keys longer than 3072 bits. Give a Key pair name to your SSH key.
30+
8. In the "Inbound port rules", select HTTP (80) and SSH (22) as the inbound ports.
31+
32+
![Azure portal VM creation — Azure Cobalt 100 Arm64 virtual machine (D4ps_v6) alt-text#center](images/instance1.png "Figure 2: Allow inbound port rules")
33+
34+
9. Click on the "Review + Create" tab and review the configuration for your virtual machine. It should look like the following:
35+
36+
![Azure portal VM creation — Azure Cobalt 100 Arm64 virtual machine (D4ps_v6) alt-text#center](images/ubuntu-pro.png "Figure 3: Review and Create an Azure Cobalt 100 Arm64 VM")
37+
38+
10. Finally, when you are confident about your selection, click on the "Create" button, and click on the "Download Private key and Create Resources" button.
39+
40+
![Azure portal VM creation — Azure Cobalt 100 Arm64 virtual machine (D4ps_v6) alt-text#center](images/instance4.png "Figure 4: Download Private key and Create Resources")
41+
42+
11. Your virtual machine should be ready and running within no time. You can SSH into the virtual machine using the private key, along with the Public IP details.
43+
44+
![Azure portal VM creation — Azure Cobalt 100 Arm64 virtual machine (D4ps_v6) alt-text#center](images/final-vm.png "Figure 5: VM deployment confirmation in Azure portal")
45+
46+
{{% notice Note %}}
47+
48+
To learn more about Arm-based virtual machine in Azure, refer to “Getting Started with Microsoft Azure” in [Get started with Arm-based cloud instances](/learning-paths/servers-and-cloud-computing/csp/azure).
49+
50+
{{% /notice %}}

0 commit comments

Comments
 (0)