diff --git a/content/learning-paths/servers-and-cloud-computing/kafka-azure/baseline.md b/content/learning-paths/servers-and-cloud-computing/kafka-azure/baseline.md index 8fd3d3cff..80fda1ed4 100644 --- a/content/learning-paths/servers-and-cloud-computing/kafka-azure/baseline.md +++ b/content/learning-paths/servers-and-cloud-computing/kafka-azure/baseline.md @@ -6,10 +6,11 @@ weight: 5 layout: learningpathall --- -## Run a Baseline test with Kafka +## Run a baseline test with Kafka After installing Apache Kafka 4.1.0 on your Azure Cobalt 100 Arm64 virtual machine, you can perform a baseline test to verify that Kafka runs correctly and that messages can be produced and consumed end-to-end. Kafka 4.1.0 introduces KRaft mode (Kafka Raft Metadata mode), which integrates the control and data planes, eliminating the need for ZooKeeper. + This simplifies deployment, reduces latency, and provides a unified, self-managed Kafka cluster architecture. To run this baseline test, open four terminal sessions: @@ -20,47 +21,48 @@ To run this baseline test, open four terminal sessions: - **Terminal 4:** Read messages as the consumer. Each terminal has a specific role, helping you verify that Kafka works end-to-end on your Arm64 VM. + ## Configure and format KRaft KRaft (Kafka Raft) mode replaces ZooKeeper by managing metadata directly within the Kafka broker. This change improves scalability, reduces external dependencies, and speeds up controller failover in distributed clusters. Before you start Kafka in KRaft mode, you need to configure the broker and initialize the storage directory. You only need to do this once for each broker. +## Edit the configuration file to update KRaft properties - ## Edit the configuration file - Open the Kafka configuration file in an editor: +Use an editor to open the Kafka configuration file at `/opt/kafka/config/server.properties`. Use `sudo` so that you can save the file. - ```console - vi /opt/kafka/config/server.properties - ``` +Ensure the following configuration entries are present for a single-node KRaft setup: -## Add or modify KRaft properties - Ensure the following configuration entries are present for a single-node KRaft setup: +```java +process.roles=controller,broker +node.id=1 +controller.quorum.voters=1@localhost:9093 +listeners=PLAINTEXT://:9092,CONTROLLER://:9093 +advertised.listeners=PLAINTEXT://localhost:9092 +log.dirs=/tmp/kraft-combined-logs +``` - ```java - process.roles=controller,broker - node.id=1 - controller.quorum.voters=1@localhost:9093 - listeners=PLAINTEXT://:9092,CONTROLLER://:9093 - advertised.listeners=PLAINTEXT://localhost:9092 - log.dirs=/tmp/kraft-combined-logs - ``` - This configuration file sets up a single Kafka server to act as both a controller (managing cluster metadata) and a broker (handling data), running in KRaft mode. It defines the node's unique ID and specifies the local host as the sole participant in the controller quorum. +This configuration file sets up a single Kafka server to act as both a controller (managing cluster metadata) and a broker (handling data), running in KRaft mode. It defines the node's unique ID and specifies the local host as the sole participant in the controller quorum. ## Format the storage directory - Format the metadata storage directory using the kafka-storage.sh tool. This initializes KRaft’s internal Raft logs with a unique cluster ID. - ```console - bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties - ``` - You should see output similar to: +Format the metadata storage directory using the kafka-storage.sh tool. This initializes KRaft’s internal Raft logs with a unique cluster ID. + +```console +bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties +``` + +You should see output similar to: + +```output +Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1. +``` - ```output - Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 4.1-IV1. - ``` - This confirms that the Kafka storage directory has been successfully formatted and that the broker is ready to start in KRaft mode. +This confirms that the Kafka storage directory has been successfully formatted and that the broker is ready to start in KRaft mode. ## Perform the baseline test + With Kafka 4.1.0 installed and configured in KRaft mode, you’re now ready to run a baseline test to verify that the Kafka broker starts correctly, topics can be created, and message flow works as expected. You’ll use multiple terminals for this test: @@ -70,21 +72,25 @@ Terminal 3: send messages (Producer) Terminal 4: read messages (Consumer) ## Terminal 1 - start Kafka broker + Start the Kafka broker (the main server process responsible for managing topics and handling messages) in KRaft mode: ```console cd /opt/kafka bin/kafka-server-start.sh config/server.properties ``` + Keep this terminal open and running. The broker process must stay active for all subsequent commands. ## Terminal 2 - create a topic + Open a new terminal window. Create a topic named test-topic-kafka, which acts as a logical channel where producers send and consumers receive messages: ```console cd /opt/kafka bin/kafka-topics.sh --create --topic test-topic-kafka --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 ``` + You should see output similar to: ```output @@ -92,6 +98,7 @@ Created topic test-topic-kafka. ``` ## Verify topic creation + List available topics to confirm that your new topic was created successfully. Run the following command: ```console @@ -110,39 +117,45 @@ If you see `test-topic-kafka` in the list, your topic was created and is ready f ```console bin/kafka-topics.sh --list --bootstrap-server localhost:9092 ``` + You should see output similar to: ```output __consumer_offsets test-topic-kafka ``` + Kafka is now running, and you’ve successfully created and verified a topic. Next, you’ll use Terminal 3 to produce messages and Terminal 4 to consume messages, completing the baseline functional test on your Arm64 environment. ## Terminal 3 - console producer (write message) + In this step, you’ll start the Kafka Producer, which publishes messages to the topic test-topic-kafka. The producer acts as the data source, sending messages to the Kafka broker. ```console cd /opt/kafka bin/kafka-console-producer.sh --topic test-topic-kafka --bootstrap-server localhost:9092 ``` -After running the command, you’ll see an empty prompt. This means the producer is ready to send data. -Type the following message and press Enter: + +After running the command, you'll see an empty prompt. This means the producer is ready to send data. Type the following message and press Enter: ```output hello from azure arm vm ``` + Each line you type is sent as a message to the Kafka topic and stored on disk by the broker. ## Terminal 4 - console consumer (read message) + Next, open another terminal and start the Kafka Consumer, which subscribes to the same topic (test-topic-kafka) and reads messages from the beginning of the log: ```console cd /opt/kafka bin/kafka-console-consumer.sh --topic test-topic-kafka --from-beginning --bootstrap-server localhost:9092 ``` -If Kafka is working correctly, you should immediately see your message `hello from azure arm vm` displayed: -You’ve now completed a full end-to-end Kafka validation test on your Azure Cobalt 100 Arm64 VM, verifying producer, broker, and consumer communication. +If Kafka is working correctly, you should immediately see your message `hello from azure arm vm` displayed. + +You've now completed a full end-to-end Kafka validation test on your Azure Cobalt 100 Arm64 VM, verifying producer, broker, and consumer communication. -Now you can proceed to benchmarking Kafka’s performance on the Azure Cobalt 100 Arm virtual machine. +Now you can proceed to benchmarking Kafka's performance on the Azure Cobalt 100 Arm virtual machine. diff --git a/content/learning-paths/servers-and-cloud-computing/kafka-azure/benchmarking.md b/content/learning-paths/servers-and-cloud-computing/kafka-azure/benchmarking.md index cc4d62813..c4597c598 100644 --- a/content/learning-paths/servers-and-cloud-computing/kafka-azure/benchmarking.md +++ b/content/learning-paths/servers-and-cloud-computing/kafka-azure/benchmarking.md @@ -10,12 +10,12 @@ layout: learningpathall Apache Kafka includes official performance testing utilities that allow you to measure throughput, latency, and end-to-end efficiency of your messaging system. These tools`kafka-producer-perf-test.sh` and `kafka-consumer-perf-test.sh` are bundled with Kafka’s standard installation and are designed for realistic performance evaluation of producers and consumers. -## Steps for Kafka Benchmarking +## Steps for Kafka benchmarking Before running the benchmarks, make sure your Kafka broker is already active in a separate terminal (as configured in the previous section). Now open two new terminal sessions; one for running the producer benchmark, and the other for the consumer benchmark. -### Terminal A - Producer Benchmark +### Terminal 1 - Producer Benchmark The Producer Performance Test measures how quickly Kafka can publish messages to a topic and reports key performance metrics such as throughput, average latency, and percentile latencies. @@ -44,7 +44,7 @@ You should see output similar to: | **Max latency** | The longest single message send time recorded. | | **50th / 95th / 99th percentiles** | Distribution of message send times. For example, 95% of messages completed under 1,184 ms in the sample output. | -### Terminal B - Consumer benchmark +### Terminal 2 - Consumer benchmark The Consumer Performance Test measures how efficiently Kafka can read and process messages from a topic. It reports metrics such as total messages consumed, data throughput, and fetch rates, helping validate overall consumer-side performance on your Azure Cobalt 100 (Arm64) VM.