Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ weight: 5
layout: learningpathall
---

## Run a Baseline test with Kafka
## Run a baseline test with Kafka

After installing Apache Kafka 4.1.0 on your Azure Cobalt 100 Arm64 virtual machine, you can perform a baseline test to verify that Kafka runs correctly and that messages can be produced and consumed end-to-end.
Kafka 4.1.0 introduces KRaft mode (Kafka Raft Metadata mode), which integrates the control and data planes, eliminating the need for ZooKeeper.

This simplifies deployment, reduces latency, and provides a unified, self-managed Kafka cluster architecture.

To run this baseline test, open four terminal sessions:
Expand All @@ -20,47 +21,48 @@ To run this baseline test, open four terminal sessions:
- **Terminal 4:** Read messages as the consumer.

Each terminal has a specific role, helping you verify that Kafka works end-to-end on your Arm64 VM.

## Configure and format KRaft

KRaft (Kafka Raft) mode replaces ZooKeeper by managing metadata directly within the Kafka broker. This change improves scalability, reduces external dependencies, and speeds up controller failover in distributed clusters.

Before you start Kafka in KRaft mode, you need to configure the broker and initialize the storage directory. You only need to do this once for each broker.

## Edit the configuration file to update KRaft properties

## Edit the configuration file
Open the Kafka configuration file in an editor:
Use an editor to open the Kafka configuration file at `/opt/kafka/config/server.properties`. Use `sudo` so that you can save the file.

```console
vi /opt/kafka/config/server.properties
```
Ensure the following configuration entries are present for a single-node KRaft setup:

## Add or modify KRaft properties
Ensure the following configuration entries are present for a single-node KRaft setup:
```java
process.roles=controller,broker
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kraft-combined-logs
```

```java
process.roles=controller,broker
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kraft-combined-logs
```
This configuration file sets up a single Kafka server to act as both a controller (managing cluster metadata) and a broker (handling data), running in KRaft mode. It defines the node's unique ID and specifies the local host as the sole participant in the controller quorum.
This configuration file sets up a single Kafka server to act as both a controller (managing cluster metadata) and a broker (handling data), running in KRaft mode. It defines the node's unique ID and specifies the local host as the sole participant in the controller quorum.

## Format the storage directory
Format the metadata storage directory using the kafka-storage.sh tool. This initializes KRaft’s internal Raft logs with a unique cluster ID.

```console
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties
```
You should see output similar to:
Format the metadata storage directory using the kafka-storage.sh tool. This initializes KRaft’s internal Raft logs with a unique cluster ID.

```console
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties
```

You should see output similar to:

```output
Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1.
```

```output
Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1.
```
This confirms that the Kafka storage directory has been successfully formatted and that the broker is ready to start in KRaft mode.
This confirms that the Kafka storage directory has been successfully formatted and that the broker is ready to start in KRaft mode.

## Perform the baseline test

With Kafka 4.1.0 installed and configured in KRaft mode, you’re now ready to run a baseline test to verify that the Kafka broker starts correctly, topics can be created, and message flow works as expected.

You’ll use multiple terminals for this test:
Expand All @@ -70,28 +72,33 @@ Terminal 3: send messages (Producer)
Terminal 4: read messages (Consumer)

## Terminal 1 - start Kafka broker

Start the Kafka broker (the main server process responsible for managing topics and handling messages) in KRaft mode:

```console
cd /opt/kafka
bin/kafka-server-start.sh config/server.properties
```

Keep this terminal open and running. The broker process must stay active for all subsequent commands.

## Terminal 2 - create a topic

Open a new terminal window. Create a topic named test-topic-kafka, which acts as a logical channel where producers send and consumers receive messages:

```console
cd /opt/kafka
bin/kafka-topics.sh --create --topic test-topic-kafka --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
```

You should see output similar to:

```output
Created topic test-topic-kafka.
```

## Verify topic creation

List available topics to confirm that your new topic was created successfully. Run the following command:

```console
Expand All @@ -110,39 +117,45 @@ If you see `test-topic-kafka` in the list, your topic was created and is ready f
```console
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
```

You should see output similar to:

```output
__consumer_offsets
test-topic-kafka
```

Kafka is now running, and you’ve successfully created and verified a topic.
Next, you’ll use Terminal 3 to produce messages and Terminal 4 to consume messages, completing the baseline functional test on your Arm64 environment.

## Terminal 3 - console producer (write message)

In this step, you’ll start the Kafka Producer, which publishes messages to the topic test-topic-kafka. The producer acts as the data source, sending messages to the Kafka broker.

```console
cd /opt/kafka
bin/kafka-console-producer.sh --topic test-topic-kafka --bootstrap-server localhost:9092
```
After running the command, you’ll see an empty prompt. This means the producer is ready to send data.
Type the following message and press Enter:

After running the command, you'll see an empty prompt. This means the producer is ready to send data. Type the following message and press Enter:

```output
hello from azure arm vm
```

Each line you type is sent as a message to the Kafka topic and stored on disk by the broker.

## Terminal 4 - console consumer (read message)

Next, open another terminal and start the Kafka Consumer, which subscribes to the same topic (test-topic-kafka) and reads messages from the beginning of the log:

```console
cd /opt/kafka
bin/kafka-console-consumer.sh --topic test-topic-kafka --from-beginning --bootstrap-server localhost:9092
```
If Kafka is working correctly, you should immediately see your message `hello from azure arm vm` displayed:

You’ve now completed a full end-to-end Kafka validation test on your Azure Cobalt 100 Arm64 VM, verifying producer, broker, and consumer communication.
If Kafka is working correctly, you should immediately see your message `hello from azure arm vm` displayed.

You've now completed a full end-to-end Kafka validation test on your Azure Cobalt 100 Arm64 VM, verifying producer, broker, and consumer communication.

Now you can proceed to benchmarking Kafkas performance on the Azure Cobalt 100 Arm virtual machine.
Now you can proceed to benchmarking Kafka's performance on the Azure Cobalt 100 Arm virtual machine.
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ layout: learningpathall

Apache Kafka includes official performance testing utilities that allow you to measure throughput, latency, and end-to-end efficiency of your messaging system. These tools`kafka-producer-perf-test.sh` and `kafka-consumer-perf-test.sh` are bundled with Kafka’s standard installation and are designed for realistic performance evaluation of producers and consumers.

## Steps for Kafka Benchmarking
## Steps for Kafka benchmarking

Before running the benchmarks, make sure your Kafka broker is already active in a separate terminal (as configured in the previous section).
Now open two new terminal sessions; one for running the producer benchmark, and the other for the consumer benchmark.

### Terminal A - Producer Benchmark
### Terminal 1 - Producer Benchmark

The Producer Performance Test measures how quickly Kafka can publish messages to a topic and reports key performance metrics such as throughput, average latency, and percentile latencies.

Expand Down Expand Up @@ -44,7 +44,7 @@ You should see output similar to:
| **Max latency** | The longest single message send time recorded. |
| **50th / 95th / 99th percentiles** | Distribution of message send times. For example, 95% of messages completed under 1,184 ms in the sample output. |

### Terminal B - Consumer benchmark
### Terminal 2 - Consumer benchmark

The Consumer Performance Test measures how efficiently Kafka can read and process messages from a topic. It reports metrics such as total messages consumed, data throughput, and fetch rates, helping validate overall consumer-side performance on your Azure Cobalt 100 (Arm64) VM.

Expand Down