Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions docker/examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,18 @@ It is also possible to use the input file to have a common set of configurations
- Assuming that `KAFKA_LOG4J_LOGGERS='property1=value1,property2=value2'` environment variable is provided to Docker container.
- `log4j.logger.property1=value1` and `log4j.logger.property2=value2` will be added to the `log4j2.yaml` file inside Docker container.

Running in SASL mode
--------------------

- SASL mode requires a JAAS configuration file to be mounted into the Docker container and referenced via the `KAFKA_OPTS` environment variable.
- Mount the directory containing the JAAS config file to `/etc/kafka/secrets` in the Docker container.
- Set `KAFKA_OPTS` to `-Djava.security.auth.login.config=/etc/kafka/secrets/<jaas_config_filename>`.
- Set `KAFKA_SASL_ENABLED_MECHANISMS` to the desired SASL mechanism (e.g. `GSSAPI` , `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`).
- Ensure `KAFKA_ADVERTISED_LISTENERS` contains a `SASL_PLAINTEXT://` or `SASL_SSL://` listener.
- For inter-broker SASL communication, set `KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL` to the desired mechanism (SASL_PLAINTEXT (or SASL_SSL)).
- The Docker image `configure` script will validate that `KAFKA_OPTS` contains the `java.security.auth.login.config` property when SASL listeners are detected.
- For more details on SASL mechanisms and configuration, refer to the [Kafka SASL authentication documentation](https://kafka.apache.org/documentation/#security_sasl).

Running in SSL mode
-------------------

Expand Down Expand Up @@ -169,6 +181,25 @@ Single Node
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties
```
- SASL_PLAINTEXT:
- Here we use SASL/PLAIN mechanism for authentication.
- A JAAS configuration file is mounted into the container via a volume mount.
- `KAFKA_OPTS` is set to point to the JAAS config file.
- Similar to the plaintext example, two listeners are configured: one for inter-broker communication and one for client-to-broker communication. Both use the `SASL_PLAINTEXT` security protocol.
- Two users are configured in the JAAS file: `admin` (for inter-broker) and `alice` (for clients).
- Note: SASL is currently not supported with the GraalVM based native image (`apache/kafka-native`) due to missing reflection configuration for `java.security.AccessController`. See [KAFKA-19584](https://issues.apache.org/jira/browse/KAFKA-19584) for details.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an open issue.

- To run the example:
```
# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/sasl_plaintext/docker-compose.yml up
```
- To produce messages using client scripts (Ensure that java version >= 17):
```
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties
```

Multi Node Cluster
------------------
Expand Down Expand Up @@ -221,6 +252,23 @@ Multi Node Cluster
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties
```
- SASL_PLAINTEXT:
- Similar to Plaintext example, with SASL/PLAIN authentication added.
- Similar to Plaintext example, two listeners are configured for inter-broker and client-to-broker communication, both using the `SASL_PLAINTEXT` security protocol. Controllers use `PLAINTEXT`.
- Each broker mounts the same JAAS config file. In production, each broker should have its own credentials.
- Note: SASL is currently not supported with the GraalVM based native image (`apache/kafka-native`) due to missing reflection configuration for `java.security.AccessController`. See [KAFKA-19584](https://issues.apache.org/jira/browse/KAFKA-19584) for details.
- To run the example:
```
# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/sasl_plaintext/docker-compose.yml up
```
- To produce messages using client scripts (Ensure that java version >= 17):
```
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties
```
- Isolated:
- Examples are present in `docker-compose-files/cluster/isolated` directory.
- Plaintext:
Expand Down Expand Up @@ -260,5 +308,21 @@ Multi Node Cluster
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties
```
- SASL_PLAINTEXT:
- Same as combined SASL_PLAINTEXT example, with controllers and brokers separated.
- `SASL_PLAINTEXT` is only for inter-broker and client communication. Controllers use `PLAINTEXT`.
- Note: SASL is currently not supported with the GraalVM based native image (`apache/kafka-native`) due to missing reflection configuration for `java.security.AccessController`. See [KAFKA-19584](https://issues.apache.org/jira/browse/KAFKA-19584) for details.
- To run the example:
```
# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/sasl_plaintext/docker-compose.yml up
```
- To produce messages using client scripts (Ensure that java version >= 17):
```
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties
```

- Note that the examples are meant to be tried one at a time, make sure you close an example server before trying out the other to avoid conflicts.
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
services:
kafka-1:
image: ${IMAGE}
hostname: kafka-1
container_name: kafka-1
ports:
- 29094:9094
volumes:
- ../../../../fixtures/sasl:/etc/kafka/secrets
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093'
KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,CONTROLLER://:9093,SASL_PLAINTEXT_HOST://:9094'
KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-1:19094,SASL_PLAINTEXT_HOST://localhost:29094
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN'
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN'
KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf'

kafka-2:
image: ${IMAGE}
hostname: kafka-2
container_name: kafka-2
ports:
- 39094:9094
volumes:
- ../../../../fixtures/sasl:/etc/kafka/secrets
environment:
KAFKA_NODE_ID: 2
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093'
KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,CONTROLLER://:9093,SASL_PLAINTEXT_HOST://:9094'
KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-2:19094,SASL_PLAINTEXT_HOST://localhost:39094
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN'
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN'
KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf'

kafka-3:
image: ${IMAGE}
hostname: kafka-3
container_name: kafka-3
ports:
- 49094:9094
volumes:
- ../../../../fixtures/sasl:/etc/kafka/secrets
environment:
KAFKA_NODE_ID: 3
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093'
KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,CONTROLLER://:9093,SASL_PLAINTEXT_HOST://:9094'
KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-3:19094,SASL_PLAINTEXT_HOST://localhost:49094
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN'
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN'
KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf'
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
services:
controller-1:
image: ${IMAGE}
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: 'controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LISTENERS: 'CONTROLLER://:9093'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'

controller-2:
image: ${IMAGE}
environment:
KAFKA_NODE_ID: 2
KAFKA_PROCESS_ROLES: 'controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LISTENERS: 'CONTROLLER://:9093'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'

controller-3:
image: ${IMAGE}
environment:
KAFKA_NODE_ID: 3
KAFKA_PROCESS_ROLES: 'controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LISTENERS: 'CONTROLLER://:9093'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'

kafka-1:
image: ${IMAGE}
hostname: kafka-1
container_name: kafka-1
ports:
- 29094:9094
volumes:
- ../../../../fixtures/sasl:/etc/kafka/secrets
environment:
KAFKA_NODE_ID: 4
KAFKA_PROCESS_ROLES: 'broker'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT'
KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,SASL_PLAINTEXT_HOST://:9094'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-1:19094,SASL_PLAINTEXT_HOST://localhost:29094
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN'
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN'
KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf'
depends_on:
- controller-1
- controller-2
- controller-3

kafka-2:
image: ${IMAGE}
hostname: kafka-2
container_name: kafka-2
ports:
- 39094:9094
volumes:
- ../../../../fixtures/sasl:/etc/kafka/secrets
environment:
KAFKA_NODE_ID: 5
KAFKA_PROCESS_ROLES: 'broker'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT'
KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,SASL_PLAINTEXT_HOST://:9094'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-2:19094,SASL_PLAINTEXT_HOST://localhost:39094
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN'
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN'
KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf'
depends_on:
- controller-1
- controller-2
- controller-3

kafka-3:
image: ${IMAGE}
hostname: kafka-3
container_name: kafka-3
ports:
- 49094:9094
volumes:
- ../../../../fixtures/sasl:/etc/kafka/secrets
environment:
KAFKA_NODE_ID: 6
KAFKA_PROCESS_ROLES: 'broker'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT'
KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,SASL_PLAINTEXT_HOST://:9094'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093'
KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-3:19094,SASL_PLAINTEXT_HOST://localhost:49094
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN'
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN'
KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf'
depends_on:
- controller-1
- controller-2
- controller-3
Loading