Skip to content

Commit 9c48f97

Browse files
shibbirmccshibbirmcc
authored
Merge pull request #9 from shibbirmcc/feature/update-kafka-documentation
docs: update KAFKA.md for testcontainer KRaft mode implementation
2 parents 8e538e7 + b0812fe commit 9c48f97

File tree

1 file changed

+132
-81
lines changed

1 file changed

+132
-81
lines changed

‎KAFKA.md‎

Lines changed: 132 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,49 @@
1-
# **Running Kakfa during local execution and tests**
1+
# **Running Kafka during local execution and tests**
22

3-
## Local Execution Kakfa Commands
3+
## Local Execution Kafka Commands
4+
5+
### Option 1: Modern KRaft Mode (Recommended)
6+
Using the official Apache Kafka image with KRaft mode (no Zookeeper required):
7+
8+
```shell
9+
# Clean up any existing containers
10+
sudo docker container prune
11+
12+
# Run Kafka in KRaft mode
13+
sudo docker run -d --name kafka \
14+
-p 9092:9092 \
15+
-e KAFKA_NODE_ID=1 \
16+
-e KAFKA_PROCESS_ROLES=broker,controller \
17+
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
18+
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \
19+
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
20+
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT \
21+
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
22+
-e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT \
23+
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
24+
-e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
25+
-e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
26+
-e KAFKA_LOG_DIRS=/tmp/kraft-combined-logs \
27+
-e KAFKA_AUTO_CREATE_TOPICS_ENABLE=true \
28+
apache/kafka:3.8.1
29+
30+
# Create topic
31+
sudo docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
32+
--create --topic credentials \
33+
--bootstrap-server localhost:9092 \
34+
--partitions 1 --replication-factor 1
35+
36+
# Console consumer
37+
sudo docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
38+
--bootstrap-server localhost:9092 --topic credentials --from-beginning
39+
40+
# Console producer
41+
sudo docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
42+
--bootstrap-server localhost:9092 --topic credentials
43+
```
44+
45+
### Option 2: Legacy Zookeeper Mode
46+
Using Confluent images with separate Zookeeper (for compatibility with older setups):
447
```shell
548
sudo docker container prune
649

@@ -31,86 +74,94 @@ sudo docker exec -it 636092f818f5 kafka-console-producer.sh --bootstrap-server l
3174
```
3275

3376
## TestContainer Execution
34-
### Create a docker image with embedded zookeeper using below Dockerfile:
35-
```dockerfile
36-
# Use a lightweight base image
37-
FROM ubuntu:20.04
38-
39-
# Install dependencies
40-
RUN apt-get update && \
41-
apt-get install -y openjdk-11-jre wget tar netcat && \
42-
apt-get clean
43-
44-
# Set Kafka and Zookeeper versions
45-
ENV KAFKA_VERSION=3.9.0
46-
ENV SCALA_VERSION=2.13
47-
ENV KAFKA_HOME=/opt/kafka
48-
49-
# Download and extract Kafka
50-
RUN wget https://downloads.apache.org/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -O /tmp/kafka.tgz && \
51-
mkdir -p ${KAFKA_HOME} && \
52-
tar -xvzf /tmp/kafka.tgz --strip-components=1 -C ${KAFKA_HOME} && \
53-
rm /tmp/kafka.tgz
54-
55-
# Add Kafka binaries to PATH
56-
ENV PATH="${KAFKA_HOME}/bin:${PATH}"
57-
58-
# Copy start script to container
59-
COPY start-kafka-zookeeper.sh /usr/bin/start-kafka-zookeeper.sh
60-
RUN chmod +x /usr/bin/start-kafka-zookeeper.sh
61-
62-
# Expose Zookeeper and Kafka ports
63-
EXPOSE 2181 9092
64-
65-
# Set environment variables
66-
ENV ZOOKEEPER_CLIENT_PORT=2181
67-
ENV KAFKA_BROKER_ID=1
68-
ENV KAFKA_ZOOKEEPER_CONNECT=localhost:2181
69-
ENV KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
70-
ENV KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
71-
ENV KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
72-
73-
# Start both services
74-
CMD ["start-kafka-zookeeper.sh"]
75-
```
7677

77-
### Create the startup script called ``start-kafka-zookeeper.sh``
78-
```shell
79-
#!/bin/bash
80-
81-
# Start Zookeeper
82-
echo "Starting Zookeeper..."
83-
${KAFKA_HOME}/bin/zookeeper-server-start.sh ${KAFKA_HOME}/config/zookeeper.properties > /var/log/zookeeper.log 2>&1 &
84-
85-
# Wait for Zookeeper to start
86-
echo "Waiting for Zookeeper to start..."
87-
while ! nc -z localhost 2181; do
88-
sleep 1
89-
done
90-
echo "Zookeeper started successfully."
91-
92-
# Start Kafka
93-
echo "Starting Kafka..."
94-
${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/server.properties > /var/log/kafka.log 2>&1 &
95-
96-
# Wait for Kafka to start
97-
echo "Waiting for Kafka to start..."
98-
while ! grep -q "started (kafka.server.KafkaServer)" /var/log/kafka.log; do
99-
sleep 1
100-
done
101-
echo "Kafka started successfully."
102-
103-
# Keep the container running
104-
tail -f /dev/null
105-
```
106-
### Create and push the image to an public repository
107-
N.B: You have to have an account to any public repository to push any custom image.
108-
```shell
109-
docker build -t <repository_username>/kafka-with-zookeeper:latest .
110-
docker push <repository_username>/kafka-with-zookeeper:latest
78+
The testcontainer implementation now uses the official Apache Kafka image with KRaft mode, eliminating the need for Zookeeper. This provides a simpler, more modern setup.
79+
80+
### Key Features:
81+
- **No Zookeeper Required**: Uses Kafka's KRaft mode (Kafka Raft metadata mode)
82+
- **Official Apache Kafka Image**: Uses `apache/kafka:3.8.1`
83+
- **Automatic Topic Creation**: Creates the required topic programmatically
84+
- **Simplified Configuration**: Fewer moving parts and dependencies
85+
86+
### TestContainer Configuration:
87+
The testcontainer is configured with the following KRaft mode settings:
88+
89+
```go
90+
kafkaReq := testcontainers.ContainerRequest{
91+
Image: "apache/kafka:3.8.1", // Latest stable Kafka image with Kraft mode
92+
ExposedPorts: []string{"9092/tcp"}, // Only Kafka port needed (no Zookeeper)
93+
Env: map[string]string{
94+
// Kraft mode configuration
95+
"KAFKA_NODE_ID": "1",
96+
"KAFKA_PROCESS_ROLES": "broker,controller",
97+
"KAFKA_CONTROLLER_QUORUM_VOTERS": "1@localhost:9093",
98+
"KAFKA_LISTENERS": "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093",
99+
"KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:9092",
100+
"KAFKA_LISTENER_SECURITY_PROTOCOL_MAP": "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT",
101+
"KAFKA_CONTROLLER_LISTENER_NAMES": "CONTROLLER",
102+
"KAFKA_INTER_BROKER_LISTENER_NAME": "PLAINTEXT",
103+
"KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR": "1",
104+
"KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR": "1",
105+
"KAFKA_TRANSACTION_STATE_LOG_MIN_ISR": "1",
106+
"KAFKA_LOG_DIRS": "/tmp/kraft-combined-logs",
107+
"KAFKA_AUTO_CREATE_TOPICS_ENABLE": "true",
108+
},
109+
WaitingFor: wait.ForLog("Kafka Server started").WithStartupTimeout(120 * time.Second),
110+
}
111111
```
112112

113-
The embedded zookeeper anf kafka image is already available publicly, just pull this image:
113+
### Benefits of KRaft Mode:
114+
1. **Simplified Architecture**: No separate Zookeeper cluster to manage
115+
2. **Better Performance**: Reduced latency and improved throughput
116+
3. **Easier Scaling**: Simpler cluster management and scaling operations
117+
4. **Official Support**: Uses the official Apache Kafka image maintained by the Kafka team
118+
5. **Future-Proof**: KRaft is the future of Kafka (Zookeeper is being deprecated)
119+
120+
### Usage in Tests:
121+
The testcontainer automatically:
122+
- Starts a Kafka broker in KRaft mode
123+
- Waits for Kafka to be fully ready
124+
- Creates the required topic specified in `KAFKA_TOPIC` environment variable
125+
- Sets the `KAFKA_BROKERS` environment variable for the application to use
126+
- Provides a teardown function to clean up resources
127+
128+
### Environment Variables Required:
129+
- `KAFKA_TOPIC`: The name of the Kafka topic to create for testing
130+
- `KAFKA_BROKERS`: Automatically set by the testcontainer setup function
131+
132+
### Manual Testing with Official Image:
133+
If you want to run Kafka manually for testing, you can use the same official image:
134+
114135
```shell
115-
docker pull shibbirmcc/kafka-with-zookeeper:latest
116-
```
136+
# Run Kafka in KRaft mode
137+
docker run -d --name kafka \
138+
-p 9092:9092 \
139+
-e KAFKA_NODE_ID=1 \
140+
-e KAFKA_PROCESS_ROLES=broker,controller \
141+
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
142+
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \
143+
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
144+
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT \
145+
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
146+
-e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT \
147+
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
148+
-e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
149+
-e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
150+
-e KAFKA_LOG_DIRS=/tmp/kraft-combined-logs \
151+
-e KAFKA_AUTO_CREATE_TOPICS_ENABLE=true \
152+
apache/kafka:3.8.1
153+
154+
# Create topic manually if needed
155+
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
156+
--create --topic credentials \
157+
--bootstrap-server localhost:9092 \
158+
--partitions 1 --replication-factor 1
159+
160+
# Console consumer
161+
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
162+
--bootstrap-server localhost:9092 --topic credentials --from-beginning
163+
164+
# Console producer
165+
docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
166+
--bootstrap-server localhost:9092 --topic credentials
167+
```

0 commit comments

Comments
 (0)