You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/kafka-azure/benchmarking.md
+47-51Lines changed: 47 additions & 51 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,19 +6,20 @@ weight: 6
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Benchmark Kafka on Azure Cobalt 100 Arm-based instances and x86_64 instances
9
+
## Benchmark Kafka on Azure Cobalt 100 Arm-based instances
10
10
11
-
Kafka’s official performance tools (**kafka-producer-perf-test.sh** and **kafka-consumer-perf-test.sh**) let you generate test workloads, measure message throughput, and record end-to-end latency.
11
+
Apache Kafka includes official performance testing utilities that allow you to measure throughput, latency, and end-to-end efficiency of your messaging system. These tools`kafka-producer-perf-test.sh` and `kafka-consumer-perf-test.sh` are bundled with Kafka’s standard installation and are designed for realistic performance evaluation of producers and consumers.
12
12
13
13
## Steps for Kafka Benchmarking
14
14
15
-
Before starting the benchmark, ensure that the **Kafka broker** are already running in separate terminals.
16
-
17
-
Now, open two new terminals—one for the **producer benchmark** and another for the **consumer benchmark**.
15
+
Before running the benchmarks, make sure your Kafka broker is already active in a separate terminal (as configured in the previous section).
16
+
Now open two new terminal sessions — one for running the producer benchmark and another for the consumer benchmark.
18
17
19
18
### Terminal A - Producer Benchmark
20
19
21
-
The producer benchmark measures how fast Kafka can send messages, reporting throughput and latency percentiles.
20
+
The Producer Performance Test measures how quickly Kafka can publish messages to a topic and reports key performance metrics such as throughput, average latency, and percentile latencies.
21
+
22
+
Run the following command to simulate message production on your Azure Cobalt 100 Arm64 VM:
22
23
23
24
```console
24
25
cd /opt/kafka
@@ -34,10 +35,20 @@ You should see output similar to:
34
35
```output
35
36
1000000 records sent, 252589.0 records/sec (24.09 MB/sec), 850.85 ms avg latency, 1219.00 ms max latency, 851 ms 50th, 1184 ms 95th, 1210 ms 99th, 1218 ms 99.9th.
|**Records/sec**| Number of messages successfully produced per second. Higher indicates better throughput. |
42
+
|**MB/sec**| Total data throughput in megabytes per second. |
43
+
|**Avg latency**| Average time (in milliseconds) for the producer to send a message and receive acknowledgment from the broker. |
44
+
|**Max latency**| The longest single message send time recorded. |
45
+
|**50th / 95th / 99th percentiles**| Distribution of message send times. For example, 95% of messages completed under 1,184 ms in the sample output. |
46
+
37
47
### Terminal B - Consumer benchmark
38
48
39
-
The consumer benchmark measures how fast Kafka can read messages from the topic, reporting throughput and total messages consumed.
49
+
The Consumer Performance Test measures how efficiently Kafka can read and process messages from a topic. It reports metrics such as total messages consumed, data throughput, and fetch rates, helping validate overall consumer-side performance on your Azure Cobalt 100 (Arm64) VM.
40
50
51
+
Run the following command in a new terminal:
41
52
```console
42
53
cd /opt/kafka
43
54
bin/kafka-consumer-perf-test.sh \
@@ -52,21 +63,21 @@ You should see output similar to:
|**`data.consumed.in.MB`**| Total data consumed during the benchmark. |
71
+
|**`MB.sec`**| Consumption throughput in megabytes per second. Higher values indicate better sustained read performance. |
72
+
|**`data.consumed.in.nMsg`**| Total number of messages successfully consumed. |
73
+
|**`nMsg.sec`**| Messages consumed per second (a key measure of consumer-side throughput). |
74
+
|**`fetch.time.ms`**| Time spent retrieving messages from the broker. Lower values mean faster message delivery. |
75
+
|**`fetch.nMsg.sec`**| Per-fetch message rate, useful for comparing network and I/O efficiency. |
76
+
|**`rebalance.time.ms`**| Time spent coordinating consumer group assignments before actual consumption begins. |
67
77
68
78
## Benchmark summary on Arm64:
69
-
Here is a summary of benchmark results collected on an Arm64 **D4ps_v6 Ubuntu Pro 24.04 LTS virtual machine**.
79
+
The following results summarize Kafka producer and consumer benchmark performance on an Azure Cobalt 100 (Arm64) virtual machine, specifically a D4ps_v6 instance running Ubuntu Pro 24.04 LTS.
80
+
These results validate Kafka’s stability and throughput consistency on Arm-based infrastructure.
| Value | 1,000,000 |242,013.6|23.08|840.69|1351.00 |832|1283|1330|1350|
102
+
| Value | 1,000,000 |257,532.8|24.56|816.19|1237.00 |799|1168|1220|1231|
108
103
| Unit | Records | Records/sec | MB/sec | ms | ms | ms | ms | ms | ms |
109
-
110
-
## Benchmark comparison insights
111
-
When comparing the results on Arm64 vs x86_64 virtual machines:
112
-
113
-
114
-
- The Kafka **consumer** achieved **25.02 MB/sec throughput**, processing ~**262K messages/sec** with fetch throughput exceeding **1.85M messages/sec**.
115
-
- The Kafka **producer** sustained **23.08 MB/sec throughput**, with an average latency of ~**841 ms** and peak latency of ~**1351 ms**.
116
-
- These results confirm stable Kafka performance on the **Azure Ubuntu Pro arm64 virtual machine**, validating its suitability for **baseline testing and benchmarking**.
117
-
118
-
You have now benchmarked Kafka on an Azure Cobalt 100 Arm64 virtual machine and compared results with x86_64.
104
+
Interpretation:
105
+
The producer sustained a throughput of ~257,500 records/sec (~24.5 MB/sec) with an average latency of 816 ms.
106
+
The 95th percentile latency (1168 ms) and 99th percentile (1220 ms) show predictable network and I/O performance.
107
+
Kafka maintained consistent throughput, even under full-speed production, with no message loss or broker errors reported.
108
+
109
+
### Benchmark Comparison Insights
110
+
When analyzing performance on Azure Cobalt 100 Arm64 virtual machines:
111
+
**Producer efficiency**: The producer reached ~23–25 MB/sec throughput with average latencies below 900 ms, demonstrating stable delivery rates for high-volume workloads.
112
+
**Consumer scalability**: The consumer maintained ~262K messages/sec throughput with near-linear scaling of fetch performance — exceeding 1.85M messages/sec internally.
113
+
**Performance stability**: Both producer and consumer benchmarks showed low jitter and consistent latency distribution across iterations, confirming Kafka’s predictable behavior on Arm-based VMs.
114
+
afka on an Azure Cobalt 100 Arm64 virtual machine and compared results with x86_64.
0 commit comments